new kernel 6.5.0-14(22.04) can not compile NVIDIA display card driver

[MAFoElffen]

@looki901 ---

Darn --- Look at this in your output:

    Linux version 6.5.0-060500-generic (kernel@kathleen)     (x86_64-linux-gnu-gcc-13 
    (Ubuntu 13.2.0-6ubuntu1) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) 
    #202308271831 SMP PREEMPT_DYNAMIC Thu Nov 16 11:17:07 UTC 2023

So do tell--> 6.5.0 Kernels from the 22.04.X HWE series were 6.5.0-14-generic & 6.5.0-15-generic. There were compiled with ggc-12. Where and how do you have kernel 6.5.0-060500-generic??? I don't see that kernel in any Ubuntu repo's.

What is the output of this please... Posted only within CODE Tags:

apt-cache policy linux-image-6.5.0-060500-generic

I'm thinking that will come back as blank, because I don't recognize it as an Ubuntu (compiled) Kernel through the normal channels. It didn't happen during a normal update for the HWE Series kernels channel. So saying "it is not fixed", is a misnomer in your case, because it falls outside of those channels. Right? If not, then you still have "something else" going on, causing you to have that OOT kernel, and then should be investigated.

If you look at the format of /proc/version, the field in the first parentheses is the User who built the kernel...

For mine:

mafoelffen@msi-ubuntu:~$ grep . /proc/version
Linux version 6.5.0-14-generic (buildd@lcy02-amd64-110) (x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2

That tells me mine were built by Ubuntu, through the normal launchpad built channels. Your's being "kernel@kathleen" tells me, that was a home-brew compiled kernel that was self-compiled outside the normal channels.

So, if I compile a kernel myself, I understand that is outside of the blame of Ubuntu... and that I am responsible for what was my own self-inflicted problem. You understand that right? This is where "third-party" pieces, make things complicated, when it comes to updates.

But following the same logic, building a kernel module for that would need the same compiler as that target kernel was compiled with, in your case gcc-13.

[looki901]

@looki901 ---

Darn --- Look at this in your output:

    Linux version 6.5.0-060500-generic (kernel@kathleen)     (x86_64-linux-gnu-gcc-13 
    (Ubuntu 13.2.0-6ubuntu1) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) 
    #202308271831 SMP PREEMPT_DYNAMIC Thu Nov 16 11:17:07 UTC 2023

So do tell--> 6.5.0 Kernels from the 22.04.X HWE series were 6.5.0-14-generic & 6.5.0-15-generic. There were compiled with ggc-12. Where and how do you have kernel 6.5.0-060500-generic??? I don't see that kernel in any Ubuntu repo's.

What is the output of this please... Posted only within CODE Tags:

apt-cache policy linux-image-6.5.0-060500-generic

I'm thinking that will come back as blank, because I don't recognize it as an Ubuntu (compiled) Kernel through the normal channels. It didn't happen during a normal update for the HWE Series kernels channel. So saying "it is not fixed", is a misnomer in your case, because it falls outside of those channels. Right? If not, then you still have "something else" going on, causing you to have that OOT kernel, and then should be investigated.

If you look at the format of /proc/version, the field in the first parentheses is the User who built the kernel...

For mine:

mafoelffen@msi-ubuntu:~$ grep . /proc/version
Linux version 6.5.0-14-generic (buildd@lcy02-amd64-110) (x86_64-linux-gnu-gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2

That tells me mine were built by Ubuntu, through the normal launchpad built channels. Your's being "kernel@kathleen" tells me, that was a home-brew compiled kernel that was self-compiled outside the normal channels.

So, if I compile a kernel myself, I understand that is outside of the blame of Ubuntu... and that I am responsible for what was my own self-inflicted problem. You understand that right? This is where "third-party" pieces, make things complicated, when it comes to updates.

But following the same logic, building a kernel module for that would need the same compiler as that target kernel was compiled with, in your case gcc-13.

You are right about the output of that command, it came out blank. And you are also right about that something is going on, and I have no idea about what is going on, I did not choose that kernel version and "source" on purpose.

I clicked to see what is the PPA from the Mainline Kernels (the app from the screenshot), and it directs me to the Ubuntu Repo (https://kernel.ubuntu.com/mainline/), and that is what I was expecting, it to be from the Ubuntu Repo, and that is why I thought the issue with the Nvidia drivers was not fixed for this kernel version.

I need some help to get the right kernel installed in my Ubuntu 22.04, and remove the current wrong one. What you would recommend?

[MAFoElffen]

So yes, you added a kernel from outside the normal channels. The Mainline PPA kernels are not Ubuntu Prep'ed, tweaked, and optimized. They also do not contained any security patches.

The Mainline kernel PPA is for testing kernels from the kernel.org git repo's. At launchpad, often times when there is a suspected kernel problem, they will recommend trying a certain version (directed) of that PPA, to see if that kernel resolves the problem...

They were never meant for everyday use by users... Especially since, like I said, they do not receive any security patches.

But in saying that, that tells them if it was something they did in their set compile options with their Ubuntu specific tweaks, or how the kernel was compiled as generic options from kernel.org. That will tell them if it is an Ubuntu problem or an upstream kernel problem.

If you do this

apt list linux-*-6.5.0-060500-* --installed

That will tell you what package names you need to use to remove them, in the format of

sudo apt remove --purge <package_name> [<other package names>]

[looki901]

Thank you for the help!

That solved the issue, and I got back to the old kernel at boot, then I used this command to get the right official kernel.

sudo apt install linux-image-generic-hwe-22.04

[father_ted]

I'm using vanilla LTS with HWE. I've started to get :
NVRM: loading NVIDIA UNIX x86_64 Kernel Module 535.154.05 Thu Dec 28 15:37:48 UTC 2023
nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 535.154.05 Thu Dec 28 15:51:29 UTC 2023
So it looks like something has been done about this in the last few updates.

[scottbomb]

If this work-around helped, then join this Bug Report as "Also Affects Me": https://ubuntuforums.org/showthread....4#post14175164

That link brings us back to this page, not a launchpad bug report.

[ubfan1]

Oops that should be https://bugs.launchpad.net/ubuntu/+s...s/+bug/2051457 for the bug report.

[MAFoElffen]

New work-around for Jammy and Mantic for nvidia-drivers-390, for Jammy and Mantic with kernels 6.5.0 and newer: https://bugs.launchpad.net/ubuntu/+s...57/comments/15

[robaol]

Hi
Thank you very much for posting so much helpful info. I am following your post #19 ubuntuforums.org/showthread.php?t=2494273&p=14175164#post14175164 .
My graphics card needs the 390 driver. I tried

sudo add-apt-repository https://launchpad.net/~dtl131/+archive/ubuntu/nvidiaexp/+packages

, which seemed to work. When I tried

sudo update

I got a warning because it was not signed, so I added

--allowUnathenticated

and then I got a files not found error. AFA you know, do think this PPA still works? Could you please tell me what should I have done to make it work?

**** I have now seen your post on how to install nvidiaexp ppa so I will give it a try. I assume there were no problems with missing signatures.

In the meantime, I followed your previous instructions, extending them to remove all the 6.5.0-nn intervening kernel versions and that has got me going, so thanks again for your clear instructions.

On a slightly wider topic, why does the compiler version used to build the kernel impact which kernel modules can be used? I could understand if there was linkage to a newer libc and dependencies on symbol signature/call argument versions, but that doesn't seem likely in this situation?

Thanks
R

If now solved: Please visit the "Thread Tools" link in the upper right of the page & mark this thread as "Solved"... So that other may find your solution to help them with resolving this problem.
....

---

Updated: For NVidia legacy driver 390 for Jammy and Mantic, use this PPA (credit Yellow Pasque): https://launchpad.net/~dtl131/+archi...aexp/+packages
Tested. Works with 6.5.0 series kernels.

Old outdated work-around
Legacy Drivers: Use 6.2.0 series kernels...

sudo apt remove --purge --yes linux-headers-6.5.0-14-generic linux-image-6.5.0-14-generic linux-modules-6.5.0-14-generic linux-modules-extra-6.5.0-14-generic
sudo apt install --reinstall --yes linux-headers-6.2.0-39-generic linux-image-6.2.0-39-generic linux-modules-6.2.0-39-generic linux-modules-extra-6.2.0-39-generic
sudo apt-mark hold linux-headers-6.2.0-39-generic linux-image-6.2.0-39-generic linux-modules-6.2.0-39-generic linux-modules-extra-6.2.0-39-generic

When they come up with a fix to compile NVidia Legacy drivers with Linux kernel 6.5 series kernels, then you can re-add that kernel series by unpinning the kernels via

sudo apt-mark unhold linux-headers-6.2.0-39-generic linux-image-6.2.0-39-generic linux-modules-6.2.0-39-generic linux-modules-extra-6.2.0-39-generic
sudo apt update && sudo apt upgrade

[MAFoElffen]

On a slightly wider topic, why does the compiler version used to build the kernel impact which kernel modules can be used? I could understand if there was linkage to a newer libc and dependencies on symbol signature/call argument versions, but that doesn't seem likely in this situation?

In the NV code, in one place it uses a compiler option that is in gcc-12, but not in gcc-11 (-ftrivial-auto-var-init=zero)... But in another area, NV code checks for the version of compiler that the kernel was compiled with...

That is documented in NVidia's Doc's for "Common Problems": https://download.nvidia.com/XFree86/...nproblems.html

Compiling the NVIDIA kernel module gives this error:

You appear to be compiling the NVIDIA kernel module with
a compiler different from the one that was used to compile
the running kernel. This may be perfectly fine, but there
are cases where this can lead to unexpected behavior and
system crashes.

If you know what you are doing and want to override this
check, you can do so by setting IGNORE_CC_MISMATCH.

In any other case, set the CC environment variable to the
name of the compiler that was used to compile the kernel.

    

You should compile the NVIDIA kernel module with the same compiler version that was 
used to compile your kernel. Some Linux kernel data structures are dependent on the 
version of gcc used to compile it; for example, in include/linux/spinlock.h:

        ...
        * Most gcc versions have a nasty bug with empty initializers.
        */
        #if (__GNUC__ > 2)
          typedef struct { } rwlock_t;
          #define RW_LOCK_UNLOCKED (rwlock_t) { }
        #else
          typedef struct { int gcc_is_buggy; } rwlock_t;
          #define RW_LOCK_UNLOCKED (rwlock_t) { 0 }
        #endif

If the kernel is compiled with gcc 2.x, but gcc 3.x is used when the kernel interface is compiled (or vice versa), the size of rwlock_t will vary, and things like ioremap will fail. To check what version of gcc was used to compile your kernel, you can examine the output of:

    % cat /proc/version

To check what version of gcc is currently in your $PATH, you can examine the output of:

    % gcc -v

You'll notice they were talking about gcc-versions gcc-2.x, and gcc-3.x... That has been their practice for a very, very, very long time. gcc-3.0 was released in 2001.