What is a bitmask and a mask? - Stack Overflow

Threadripper KVM GPU Passthru: Testers needed

TL;DR: Check update 8 at the bottom of this post for a fix if you don't care about the history of this issue.
For a while now it has been apparent that PCI GPU passthrough using VFIO-PCI and KVM on Threadripper is a bit broken.
This manifests itself in a number of ways: When starting a VM with a passthru GPU it will either crash or run extremely slowly without the GPU ever actually working inside the VM. Also, once a VM has been booted the output of lspci on the host changes from one kind of output to another. Finally the output of dmesg suggests an issue bringing the GPU up from D0 to D3 power state.
An example of this lspci before and after VM start, as well as dmesg kernel buffer output is included here for the 7800GTX:
08:00.0 VGA compatible controller: NVIDIA Corporation G70 [GeForce 7800 GTX] (rev a1) (prog-if 00 [VGA controller]) [ 121.409329] virbr0: port 1(vnet0) entered blocking state [ 121.409331] virbr0: port 1(vnet0) entered disabled state [ 121.409506] device vnet0 entered promiscuous mode [ 121.409872] virbr0: port 1(vnet0) entered blocking state [ 121.409874] virbr0: port 1(vnet0) entered listening state [ 122.522782] vfio-pci 0000:08:00.0: enabling device (0000 -> 0003) [ 123.613290] virbr0: port 1(vnet0) entered learning state [ 123.795760] vfio_bar_restore: 0000:08:00.0 reset recovery - restoring bars ... [ 129.534332] vfio-pci 0000:08:00.0: Refused to change power state, currently in D3 08:00.0 VGA compatible controller [0300]: NVIDIA Corporation G70 [GeForce 7800 GTX] [10de:0091] (rev ff) (prog-if ff) !!! Unknown header type 7f Kernel driver in use: vfio-pci 
Notice that lspci reports revision FF and can no longer read the header type correctly. Testing revealed that pretty much all graphics cards except Vega would exhibit this behavior, and indeed the output is very similar to the above.
Reddit user wendelltron and others suggested that the D0->D3 transition was to blame. After having gone through a brute-force exhaustive search of the BIOS, kernel and vfio-pci settings for power state transitions it is safe to assume that this is probably not the case since none of it helped.
AMD representative AMD_Robert suggested that only GPUs with EFI-compatible BIOS should be able to be used for passthru in an EFI environment, however, testing with a modern 1080GTX with EFI bios support failed in a similar way:
42:00.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1) and then 42:00.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev ff) (prog-if ff) !!! Unknown header type 7f 
Common to all the cards was that they would be unavailable in any way until the host system had been restarted. Any attempt at reading any register or configuration from the card would result in all-1 bits (or FF bytes). The bitmask used for the headers may in fact be what is causing the 7f header type (and not an actual header being read from the card). Not even physically unplugging and re-plugging the card, rescanning the PCIe bus (with /sys/bus/pci/rescan) would trigger any hotplug events or update the card info. Similarly, starting the system without the card and plugging it in would not be reflected in the PCIe bus enumeration. Some cards, once crashed, would show spurious PCIe ACS/AER errors, suggesting an issue with the PCIe controller and/or the card itself. Furthermore, the host OS would be unable to properly shut down or reboot as the kernel would hang when everything else was shut down.
A complete dissection of the vfio-pci kernel module allowed further insight into the issue. Stepping through VM initialization one line at a time (yes this took a while) it became clear that the D3 power issue may be a product of the FF register issue and that the actual instruction that kills the card may have happened earlier in the process. Specifically, the function drivers/vfio/pci/vfio_pci.c:vfio_pci_ioctl, which handles requests from userspace, has entries for VFIO_DEVICE_GET_PCI_HOT_RESET_INFO and VFIO_DEVICE_PCI_HOT_RESET and the following line of code is exactly where the cards go from active to "disconnected" states:
if (!ret) /* User has access, do the reset */ ret = slot ? pci_try_reset_slot(vdev->pdev->slot) : pci_try_reset_bus(vdev->pdev->bus); 
Commenting out this line allows the VM to boot and the GPU driver to install. Unfortunately for the nVidia cards my testing stopped here as the driver would report the well known error 43/48 for which they should be ashamed and shunned by the community. For AMD cards a R9 270 was acquired for further testing.
The reason this line is in vfio-pci is because VMs do not like getting an already initialized GPU during boot. This is a well-known problem with a number of other solutions available. By disabling the line it is neccessary to use one of the other solutions when restarting a VM. For Windows you can disable the device in Device Manager before reboot/shutdown and re-enable it again after the restart - or use login/logoff scripts to have the OS do it automatically.
Unfortunately another issue surfaced which made it clear that the VMs could only be stopped once even though they could now be rebooted many times. Once they were shut down the cards would again go into the all FF "disconnect" state. Further dissection of vfio-pci revealed another instance where an attempt to reset the slot that the GPU is in was made: in drivers/vfio/pci/vfio_pci.c:vfio_pci_try_bus_reset
if (needs_reset) ret = slot ? pci_try_reset_slot(vdev->pdev->slot) : pci_try_reset_bus(vdev->pdev->bus); 
When this line is instead skipped, a VM that has had its GPU properly disabled via Device Manager and has been properly shutdown is able to be re-launched or have another VM using the same GPU launched and works as expected.
I do not understand the underlying cause of the actual issue but the workaround seems to work with no issues except the annoyance of having to disable/re-enable the GPU from within the guest (like in ye olde days). Only speculation can be given to the real reason of this fault; the hot-reset info gathered by the ioctl may be wrong, but the ACS/AER errors suggest that the issue may be deeper in the system - perhaps the PCIe controller does not properly re-initialize the link after hot-reset just as it (or the kernel?) doesn't seem to detect hot-plug events properly even though acpihp supposedly should do that in this setup.
Here is a "screenshot" of Windows 10 running the Unigine Valley benchmark inside a VM with a Linux Mint host using KVM on Threadripper 1950x and an R9 270 passed through on an Asrock X399 Taichi with 1080GTX as host GPU:
https://imgur.com/a/0HggN
This is the culmination of many weeks of debugging. It is interesting to hear if anyone else is able to reproduce the workaround and can confirm the results. If more people can confirm this then we are one step closer to fixing the actual issue.
If you are interested in buying me a pizza, you can do so by throwing some Bitcoin in this direction: 1KToxJns2ohhX7AMTRrNtvzZJsRtwvsppx
Also, English is not my native language so feel free to ask if something was unclear or did not make any sense.
Update 1 - 2017-12-05:
Expanded search to non-gpu cards and deeper into the system. Taking memory snapshots of pcie bus for each step and comparing to expected values. Seem to have found something that may be the root cause of the issue. Working on getting documentation and creating a test to see if this is indeed the main problem and to figure out if it is a "feature" or a bug. Not allowing myself to be optimistic yet but it looks interesting, it looks fixable at multiple levels.
Update 2 - 2017-12-07:
Getting a bit closer to the real issue. The issue seems to be that KVM performs a bus reset on the secondary side of the pcie bridge above the GPU being passed through. When this happens there is an unintended side effect that the bridge changes its state somehow. It does not return in a useful configuration as you would expect and any attempt to access the GPU below it results in errors.
Manually storing the bridge 4k configuration space before the bus reset and restoring it immediately after the bus reset seems to magically bring the bridge into the expected configuration and passthru works.
The issue could probably be fixed in firmware but I'm trying to find out what part of the configuration space is fixing the issue and causing the bridge to start working again. With that information it will be possible to write a targeted patch for this quirk.
Update 3 - 2017-12-10:
Begun further isolation of what particular registers in the config space are affected unintentionally by the secondary bus reset on the bridge. This is difficult work because the changes are seemingly invisible to the kernel, they happen only in the hardware.
So far at least registers 0x19 (secondary bus number) and 0x1a (subordinate bus number) are out of sync with the values in the config space. When a bridge is in faulty mode, writing their already existing value back to them brings the bridge back into working mode.
Update 4 - 2017-12-11 ("the ugly patch"):
After looking at the config space and trying to figure out what bytes to restore from before the reset and what bytes to set to something new it became clear that this would be very difficult without knowing more about the bridge.
Instead a different strategy was followed: Ask the bridge about its current config after reset and then set its current config to what it already is; byte by byte. This brings the config space and the bridge back in sync and everything, including reset/reboot/shutdown/relaunch without scripts inside the VM, now seems to work with the cards acquired for testing. Here is the ugly patch for the brave souls who want to help test it.
Please, if you already tested the workaround: revert your changes and confirm that the bug still exists before testing this new ugly patch:
In /drivers/pci/pci.c, replace the function pci_reset_secondary_bus with this alternate version that adds the ugly patch and two variables required for it to work:
void pci_reset_secondary_bus(struct pci_dev *dev) { u16 ctrl; int i; u8 mem; pci_read_config_word(dev, PCI_BRIDGE_CONTROL, &ctrl); ctrl |= PCI_BRIDGE_CTL_BUS_RESET; pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); /* * PCI spec v3.0 7.6.4.2 requires minimum Trst of 1ms. Double * this to 2ms to ensure that we meet the minimum requirement. */ msleep(2); ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET; pci_write_config_word(dev, PCI_BRIDGE_CONTROL, ctrl); // The ugly patch for (i = 0; i < 4096; i++){ pci_read_config_byte(dev, i, &mem); pci_write_config_byte(dev, i, mem); } /* * Trhfa for conventional PCI is 2^25 clock cycles. * Assuming a minimum 33MHz clock this results in a 1s * delay before we can consider subordinate devices to * be re-initialized. PCIe has some ways to shorten this, * but we don't make use of them yet. */ ssleep(1); } 
The idea is to confirm that this ugly patch works and then beautify it, have it accepted into the kernel and to also deliver technical details to AMD to have it fixed in BIOS firmware.
Update 5 - 2017-12-20:
Not dead yet!
Primarily working on communicating the issue to AMD. This is slowed by the holiday season setting in. Their feedback could potentially help make the patch a lot more acceptable and a lot less ugly.
Update 6 - 2018-01-03 ("the java hack"):
AMD has gone into some kind of ninja mode and has not provided any feedback on the issue yet.
Due to popular demand a userland fix that does not require recompiling the kernel was made. It is a small program that runs as any user with read/write access to sysfs (this small guide assumes "root"). The program monitors any PCIe device that is connected to VFIO-PCI when the program starts, if the device disconnects due to the issues described in this post then the program tries to re-connect the device by rewriting the bridge configuration.
This program pokes bytes into the PCIe bus. Run this at your own risk!
Guide on how to get the program:
If you have any PCI devices using VFIO-PCI the program will output something along the lines of this:
------------------------------------------- Zen PCIe-Bridge BAConfig Recovery Tool, rev 1, 2018, HyenaCheeseHeads ------------------------------------------- Wed Jan 03 21:40:30 CET 2018: Detecting VFIO-PCI devices Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:40/0000:40:01.3/0000:42:00.0 Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:40/0000:40:01.3 Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.1 Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:00/0000:00:01.3 Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:40/0000:40:01.3/0000:42:00.1 Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:40/0000:40:01.3 Wed Jan 03 21:40:30 CET 2018: Device: /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.0 Wed Jan 03 21:40:30 CET 2018: Bridge: /sys/devices/pci0000:00/0000:00:01.3 Wed Jan 03 21:40:30 CET 2018: Monitoring 4 device(s)... 
And upon detecting a bridge failure it will look like this:
Wed Jan 03 21:40:40 CET 2018: Lost contact with /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.1 Wed Jan 03 21:40:40 CET 2018: Recovering 512 bytes Wed Jan 03 21:40:40 CET 2018: Bridge config write complete Wed Jan 03 21:40:40 CET 2018: Recovered bridge secondary bus Wed Jan 03 21:40:40 CET 2018: Re-acquired contact with /sys/devices/pci0000:00/0000:00:01.3/0000:08:00.1 
This is not a perfect solution but it is a stopgap measure that should allow people who do not like compiling kernels to experiment with passthru on Threadripper until AMD reacts in some way. Please report back your experience, I'll try to update the program if there are any issues with it.
Update 7 - 2018-07-10 ("the real BIOS fix"):
Along with the upcoming A.G.E.S.A. update aptly named "ThreadRipperPI-SP3r2 1.0.0.6" comes a very welcome change to the on-die PCIe controller firmware. Some board vendors have already released BETA BIOS updates with it and it will be generally available fairly soon it seems.
Initial tests on a Linux 4.15.0-22 kernel now show PCIe passthru working phenomenally!
With this change it should no longer be necessary to use any of the ugly hacks from previous updates of this thread, although they will be left here for archival reasons.
Update 8 - 2018-07-25 ("Solved for everyone?"):
Most board vendors are now pushing out official (non-BETA) BIOS updates with AGESA "ThreadRipperPI-SP3r2 1.1.0.0" including the proper fix for this issue. After updating you no longer need to use any of the temporary fixes from this thread. The BIOS updates comes as part of the preparations for supporting the Threadripper 2 CPUs which are due to be released in a few weeks from now.
Many boards support updating over the internet directly from BIOS, but in case you are a bit old-fashioned here are the links (please double-check that I linked you the right place before flashing):
Vendor Board Update Link
Asrock X399 Taichi Update to 2.3, then 3.1
Asrock X399M Taichi Update to 1.10 then 3.1
Asrock X399 Fatality Profesional Gaming Update to 2.1 then 3.1
Gigabyte X399 AURUS Gaming 7 r1 Update to F10
Gigabyte X399 DESIGNARE EX r1 Update to F10
Asus PRIME X399-A Possibly fixed in 0601 (TR2 support and sure fix inbound soon)
Asus X399 RoG Zenith Extreme Possibly fixed in 0601 (TR2 support and sure fix inbound soon)
Asus RoG Strix X399-E Gaming Possibly fixed in 0601 (TR2 support and sure fix inbound soon)
MSI X399 Gaming Pro Carbon AC Update to Beta BIOS 7B09v186 (TR2 update inbound soon)
MSI X399 SLI plus Update to Beta BIOS 7B09vA35 (TR2 update inbound soon)
submitted by HyenaCheeseHeads to Amd [link] [comments]

ICO SKYLLZ. Обзор стартапа - YouTube DigiByte (DGB) Facts Top 20! New Dogecoin Cloud Mining Site l Reviews l 130%After 24 Hours Minimum Deposit 300 l 1000 Doge Invest

It looks like you are misusing an enum as just a more cumbersome way to write a ulong.. If the days of the week are meant to be treated as a set rather than a single value (i.e., more like checkboxes than radio buttons), then you should be using the [Flags] attribute. In fact, this is almost exactly the example given in the MSDN documentation: [Flags] enum Days2 { None = 0x0, Sunday = 0x1 ... Briefly bitmask helps to manipulate position of multiple values. There is a good example here ; Bitflags are a method of storing multiple values, which are not mutually exclusive, in one variable. You've probably seen them before. Each flag is a bit position which can be set on or off. You then have a bunch of bitmasks #defined for each bit ... Bitcoin Satoshi Ethereum Blockchain Altcoin. Bitcoin. What Is Bitcoin Backed By? September 14, 2020. Bitcoin. Female student blackmailed for Bitcoin – The GW… September 14, 2020. Bitcoin. Bitcoin and Ethereum – Weekly Technical Analysis –… September 14, 2020. Bitcoin. How Bitcoin Correlations Drive the Narrative. September 13, 2020. Bitcoin. The Tell Tale Signs of a Scam Crypto ... For the coinbase_branch merkle branch, because the coinbase transaction is the first transaction in the block (if using Bitcoin as a parent chain, i.e. hash #7 in the example given below), the branch_side_mask is always going to be all zeroes, because the branch hashes will always be "on the right" of the working hash.. When only working on one auxiliary blockchain, the blockchain_branch link ... It is just a number, as represented in binary. For example, let's say I have 8 boolean values (true or false) that I want to store.I could store it as an array of 8 booleans, or I could store it as a single byte (8 bits), each of which store one of the booleans (0 = false, 1 = true).At this point, I can easily manipulate my byte so that I can (1) set specific bits to be on or off (true or ...

[index] [38836] [17302] [26706] [49437] [51093] [197] [50144] [24740] [3851] [40102]

ICO SKYLLZ. Обзор стартапа - YouTube

Обзор ICO стартапа Skyllz #icoSkyllz Разберем идею и концепцию данного стартапа, задача компании создать Единый ... Top 3 Altcoins Set To Explode in 2020 ... Altcoin Daily 39,100 views. New; 14:59. DigiByte (DGB) Jared Tate Creates BitMask A Reusable and Washable Face Mask! - Duration: 11:26. Crypto Talk 387 ... Dogestill Reviews l 130%After 24 Hours Minimum Deposit 300 Doge Instant Withdraw Running Days 0 l Plus 50.00 Dogecoin Giveaway Send ! Hindi Urdu By Crypto Bhaii Website Link https://bit.ly/327NLcs ...

#