Testers for inxi/pinxi redone -C CPU logic... huge internal changes

h2-1 · 12-13-2021, 11:44 PM

It could be something along those lines, to me it looks like a genuine kernel regression.

I believe modern amds use acpi-cpufreq, this Arch wiki article covers it reasonably well: https://wiki.archlinux.org/title/CPU_frequency_scaling

Xeon uses this, and that's about 20ms per thread/core: scaling: driver: intel_pstate governor: powersave,performance

You can assign which driver to use. Epyc system uses this: driver: acpi-cpufreq governor: ondemand

I wonder if it's caused by something the kernel is doing with certain generation of cpus, that is not intended behavior. I think this is a kernel bug.

The difference between the intel_pstate and the acpi-cpufreq driver amd uses seems to be that on intel it's a roughly 20ms delay per request, on epyc about 14ms, and ryzen, about 12-13 ms. All the intels I've tested on have been around 20ms.

2012 intel core i5 MT 2 core takes similar, 51 ms to read those 4 threads, that is, about 12ms per thread. That one doesn't have cpuinfo_cur_freq so I couldn't compare it. That's running intel_cpufreq driver, which I believe is the older version for intel.

Same issues, if what you say about how scaling_cur_freq gets its data, this really sounds like a kernel or driver bug, but given all the drivers are doing it in my tests, on all newer systems, I have to suspect this is kernel caused, not driver, or that they accidentally changed something in the kernel without realizing it would have this impact on read speeds.

Might be reason for an issue report to the kernel I have to say.

baumei · 12-14-2021, 07:35 AM

Hi "h2-1",

I also think it appears to be a bug in the kernel or something closely related to the kernel. Whoever is informed of this is apparent bug is going to need information, such as under what conditions does it occur, and when not...

Code:

occur Kernel   Processor     miti.  scaling drv.   owner
no    4.4.276  Core2 Q8300   some   acpi-cpufreq   baumei
no    4.4.276  Atom N2600    none   acpi-cpufreq   baumei
no    4.4.276  Atom N270     none   acpi-cpufreq   baumei
yes   ?        Xeon G. 6238  ?      intel-pstate   drumz
yes   ?        Epyc 7281     ?      acpi-cpufreq   h2-1
yes   ?        Ryzen 5 2600  ?      ?              h2-1
yes   ?        Core i5 x?x?  ?      intel-cpufreq  ?

I am wondering if the problem only occurs in some versions of the kernel...

h2-1 · 12-14-2021, 12:27 PM

baumei, yes, I realized we'd forgotten to collect data on kernel version.

6-core AMD Ryzen 5 2600 (MT):
12 threads: real 0m0.153s
5.14.0-18.1-liquorix-amd64
scaling: driver: acpi-cpufreq governor: ondemand

2x 6-core Intel Xeon E5-2620 0 (MT disabled):
12 threads: real ~0m0.240s
4.15.0-163-generic (ubuntu lts)

2x 16-core AMD EPYC 7281 (MT)
64 threads: real 0m0.841s
5.12-fw3

2x 4-core Intel Xeon E5-2603 (MT disabled)
8 threads: real 0m0.165s
4.15.0-163-generic (ubuntu lts)
scaling: driver: intel_pstate governor: powersave

quad core AMD GX-412TC SOC
4 threads: real 0m0.116s
5.10.0-9-amd64
scaling: driver: acpi-cpufreq governor: schedutil

2x 6-core Intel Xeon E5-2630 v2 (MT disabled)
12 threads: real ~0.260
5.4.0-91-generic (ubuntu lts)
scaling: driver: intel_pstate governor: powersave

dual core Intel Core i5-2520M (MT)
4 threads: real 0m0.064s
5.15.4-fwi
scaling: driver: intel_cpufreq governor: ondemand

2x 8-core Intel Xeon E5-2620 v4 (MT)
32 threads: real 0m0.671s
4.15.0-161-generic
scaling: driver: intel_pstate governor: powersave

As you can see, it always happens on any modern cpu, modern going back to Intel Core. Slowest was also the slowest cpu, the amd soc, at 29ms per core. Intels average 20ms. AMD zen 12-14ms.

Note most of the servers are running ubuntu lts, which get kernel patches without changing the main version number, so it's hard to know what version those are really equivalent to.

It's quite likely all these have bits of 5.x kernel running in them. So that might be related.

Almost definitely a regression in the kernel I'd guess at this point. Driver or governor doesn't seem to matter.

h2-1 · 12-14-2021, 12:44 PM

inxi 3.3.10 is now released.

Any further issues or discoveries can go towards 3.3.11. I dabbled with the idea of making this 3.4.00 or even 4.0.00 due to the immense amount of changes, but I have to follow what the README says on github, major version numbers only on primary new features, new line items that is, like CPU, Bluetooth, etc. So this will be the biggest ever point release. Will need to see how certain features do in the wild, Manjaro tends to package this very quickly so should start seeing regular user output in a few days.

The scaling_cur_freq thing is I think maybe worth following up on, would be useful to find if there is a specific kernel version where it started happening, that help nail down the regression, I hope at least it's a regression and not intended behavior.

Huge thanks for all the help, invaluable, with something this hardware focused, impossible to do any sort of even halfway decent job without it.

fourtysixandtwo · 12-15-2021, 02:48 AM

Quote:

Originally Posted by h2-1

fourtysixandtwo, it always amazes me that anything at all from OSX works with inxi/pinxi, they change stuff pretty often, and randomly just make up other stuff in terms of how they format their data sources, etc, so it's always a surprise to see inxi/pinxi run on osx without errors. Errors as in perl errors like undefined value errors are in general the only osx errors I'll spend time on, at least until Apple Corporation shells out the big bucks and pays me.

Only reason the apple stuff works at all is a guy in Finland gave me ssh access to his apple for a while many years ago, so I could get basic apple support working, but I have largely lost interest in doing anymore than just fixing perl errors for osx now, time is not infinite, nor do I have any desire to work with non free operating systems, particularly not ones that have no tools to speak of built in to work with.

It looks to me like the old osx stuff I had working is still working for cpu, that changes often, so that's nice to see it still works. FreeBSD also has no family/model/stepping data by the way. I may be getting one of those trashcan apples to play with so maybe I'll poke around apple some more, though I don't want to waste/spend/consume too much if any time on that in general. I wonder if linux works on the trashcan? Maybe I'll get an ssd and just try, though apple probably has done something to block their hardware I'd expect.

Yeah, Apple always has to be different no matter what. When I bought that machine, nothing compared hardware wise imho and the advantages far outweighed the annoyances. It was a great workhorse for my photography hobby and portable. Upgraded it to SSD about 8 years ago. I plan to put slackware on it but it's full of photos and media for kodi when I'm not home so I need to make room first.

If you do get one to play with google dosdude1 if you need to put a newer OSX on it that's not officially supported. For linux you'll want to put a boot manager on it like rEFInd.

Btw, I should have my SUN Ultra 2 up and running again in the next day or so for some inxi testing. I finally dusted it off for the first time in over 10 years and the nvram/battery chip is dead. Ordered up a replacement from digikey and some parts to fix the p90 as well. I just hope the Type 5c keyboard not completely working is a result of the nvram issue. After that I'd like to install slackware on it too.

FTIO · 12-16-2021, 09:26 AM

Sorry, gang, my stupidity is coming through with flying colors this morning...how do I install 'pinxi'? I have 'inxi', after having taken the 'build' I have for my 14.2 system, and somehow I have 'pinxi' on that system also, but I can't find the 'build' for it, so I have no clue how I got it on that system.

I'd really like to have it on this 'current' system, if possible. I even tried the instructions on the git page of:

Code:

wget -O pinxi https://github.com/smxi/inxi/raw/inxi-perl/pinxi

and at the end of that it shows 'pinxi' installed, but for the life of me I can't 'locate' or 'find' it anywhere, though if I remember right 'locate' said it was at /root/pinxi which there is not one of when I look.

Any 'builds' for pinxi anywhere I can use for 'current'?

drumz · 12-16-2021, 09:31 AM

Quote:

Originally Posted by FTIO

Sorry, gang, my stupidity is coming through with flying colors this morning...how do I install 'pinxi'? I have 'inxi', after having taken the 'build' I have for my 14.2 system, and somehow I have 'pinxi' on that system also, but I can't find the 'build' for it, so I have no clue how I got it on that system.

I'd really like to have it on this 'current' system, if possible. I even tried the instructions on the git page of:

Code:

wget -O pinxi https://github.com/smxi/inxi/raw/inxi-perl/pinxi

and at the end of that it shows 'pinxi' installed, but for the life of me I can't 'locate' or 'find' it anywhere, though if I remember right 'locate' said it was at /root/pinxi which there is not one of when I look.

Any 'builds' for pinxi anywhere I can use for 'current'?

Go back to the very first post of this thread...

Code:

# if you already have pinx installed:
pinxi -U
# if you don't, just do:
cd /usr/local/bin && sudo wget -O pinxi smxi.org/pinxi && sudo chmod +x pinxi

Modify to taste (i.e., you may not want pinxi in /usr/local/bin/).

drumz · 12-16-2021, 09:33 AM

Quote:

Originally Posted by FTIO

Code:

wget -O pinxi https://github.com/smxi/inxi/raw/inxi-perl/pinxi

and at the end of that it shows 'pinxi' installed

wget downloads to the current directory. Did you look there?

mlangdn · 12-16-2021, 11:55 AM

pinxi is a perl script. Run it by issuing

Code:

./pinxi -MCazy1

as an example. You can upgrade pinxi easily by doing

Code:

pinxi -U

h2-1 · 12-16-2021, 01:33 PM

This is by the way why I recommend installing pinxi to a directory in PATH, misplacing the downloaded file, or forgetting where I downloaded it to by accident. This is in general why I tend to use wget -O [path/filename] or curl -o [path/filename] when downloading stuff, unless I am sure I want the file in the current directory.

As an aside, I generally make pinxi user writable so I don't have to sudo or root pinxi -U when it's in /usr/local/bin since I"m updating it all the time.

There were two bugs in 3.3.10 by the way, one will hit many laptop users, the other will hit only corner cases, both are corrected in pinxi already but I'm waiting a few hours to do 3.3.11 to make sure nothing else shows up, but I think the amount of testing done here caught most of them already in 3.3.10, these were just ones where in one case, I forgot to test one specific hardware scenario, and the other was just a typo that only manifests in one case in laptop batteries, but when it does, it sprays out lot of errors.

I tossed in another -Ca feature, to show scaling_min/max_freq if either is different from cpuinf_min/max_freq values, which they can be, just so 3.3.11 wouldn't be a totally bug fix only release. I haven't found a system yet that has different min/max values, but I know those can be assigned per core if you want, that just doesn't seem to be done very often. Same as different governors can be assigned per core, I believe anyway, I've seen one server that does that, 2 governors are used on each physical cpu, on 2 blocks of cores, I assume to optimize for different behaviors.

For pinxi, _normal_ is after next inxi sync/release, pinxi is pretty much the same as inxi until changes start happening, in this case, it split again almost immediately as soon as I became aware of the bugs, which fortunately manifested almost immediately, but unfortunately, immediately after 3.3.10 went out, but objectively, in at least one case, it manifested because 3.3.10 went out and started being used. When all goes well, sometimes pinxi doesn't change much for a while after next inxi is released, but with a changeset this huge, there was no way I was going to get a bug free release right out of the box, so I'm glad the bugs were relatively minor.

h2-1 · 12-16-2021, 03:03 PM

FTIO:

Quote:

Any 'builds' for pinxi anywhere I can use for 'current'?

Note that if pinxi is used for packaging, first, it has to always be the current actual pinxi, which is generally impossible to do. If it's being used to just get the latest for 'current', that doesn't work super well because of this [top of file]:

Code:

my $self_name='pinxi';
my $self_version='3.3.10';
my $self_date='2021-12-16';
my $self_patch='02';

First, the $self_name and file name has to be changed to 'inxi', second, the $self_version is not going to be right, the data will be right, and the self patch is the current patch number of pinxi, not inxi. pinxi by definition will always be one or more patch versions greater than inxi 00 patch, and pinxi's version number will always be the same as current master inxi.

inxi, on the other hand, looks like this:

Code:

my $self_name='inxi';
my $self_version='3.3.10';
my $self_date='2021-12-13';
my $self_patch='00';

However, I'm going to do the new inxi this afternoon, just wanted to make sure no other issues were detected. In general I don't recommend packaging pinxi, first, because it can have bugs in it, along with bug fixes, and stuff that might be halfway working, or maybe which failed as an idea, and will be removed, second, because unless it's packaged essentially the same time as pinxi versions are released/committed, it will always be out of date. Arch linux gets around this issue by having in AUR a package that just grabs with git pinxi from github, which to me is somewhat pointless since that just adds a lot of complexity and bandwidth to what is otherwise a simple download of a file, or two, if you use the --man option.

The --man/--no-man options I seem to remember were added specifically (at the request of slackware forum members some time ago, and a good request it was) to avoid either downloading the man with -U, or to force the download when it doesn't happen by default. pinxi -U by default does not download the man, --man forces that. inxi -U always defaults to downloading the new man when it updates itself.

I'm not positive what is being asked here however.

FTIO · 12-16-2021, 04:02 PM

It's all good...the posts answering me helped and it's now installed alongside inxi. It was the wget thing that worked. Originally I just su'd into root and did the wget thing and I have no clue where I was in the file system. This time I made sure I was in /usr/local/bin and then did the chmod on it so my regular 'user' could also use it.

Thanks everyone for taking the time to help out a dummy. I just wanted to have it installed to be able to help the dev if I could in any way.

FTIO · 12-16-2021, 04:07 PM

Code:

user@users_place:~$ pinxi -MCazy1
Machine:
  Type: Desktop
  System: Micro-Star
    product: MS-7C35
      v: 2.0
      serial: <superuser required>
  Mobo: Micro-Star
    model: MEG X570 UNIFY (MS-7C35)
      v: 2.0
      serial: <superuser required>
  UEFI: American Megatrends LLC.
    v: A.90
    date: 05/17/2021

CPU:
  Info:
    model: AMD Ryzen 7 3700X
    bits: 64
    type: MT MCP
    arch: Zen 2
    family: 0x17 (23)
    model-id: 0x71 (113)
    stepping: 0
    microcode: 0x8701021
  Topology:
    cpus: 1
      cores: 8
        tpc: 2
      threads: 16
    smt: enabled
    cache:
      L1: 512 KiB
        desc: d-8x32 KiB; i-8x32 KiB
      L2: 4 MiB
        desc: 8x512 KiB
      L3: 32 MiB
        desc: 2x16 MiB
  Speed (MHz):
    avg: 4118
    high: 4161
    min/max: 2200/4426
    boost: enabled
    scaling:
      driver: acpi-cpufreq
      governor: ondemand
    cores:
      1: 4100
      2: 4087
      3: 4126
      4: 4138
      5: 4118
      6: 4122
      7: 4123
      8: 4129
      9: 4123
      10: 4161
      11: 4103
      12: 4110
      13: 4123
      14: 4113
      15: 4102
      16: 4117
    bogomips: 115204
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
  Vulnerabilities:
    Type: itlb_multihit
      status: Not affected
    Type: l1tf
      status: Not affected
    Type: mds
      status: Not affected
    Type: meltdown
      status: Not affected
    Type: spec_store_bypass
      mitigation: Speculative Store Bypass disabled via prctl and seccomp
    Type: spectre_v1
      mitigation: usercopy/swapgs barriers and __user pointer sanitization
    Type: spectre_v2
      mitigation: Full AMD retpoline, IBPB: conditional, STIBP: conditional, RSB filling
    Type: srbds
      status: Not affected
    Type: tsx_async_abort
      status: Not affected

h2-1 · 12-16-2021, 06:07 PM

I figured out why nobody saw the battery bug, it wasn't in the primary main battery type output, it was in the battery powered device (like mouse) output, and I never use battery powered peripherals so I never noticed it. Was scratching my head about how that one could have been missed. That's an old bug, nothing related to the 3.3.10 changes.

h2-1 · 12-16-2021, 06:32 PM

3.3.11 is now out, a fast bug fix release, almost nothing changed, except handling some corner case bugs that had slipped anyone's notice with some very specific hardware configurations.

Also adds support in case scaling_min/max_freq are different than the default physical cpuinfo_min/max_freq to show under the scaling: sub report the scaling sc-min-max: speeds, that could be useful to know too. I have not found any instances where they were different, but they can be if you set them to be different than the physical cpu limits. I don't actually know if you can change those values per core or if it's a global change.