#StackBounty: #python #docker #scrapy #scrapy-splash #windows-server-2019 Connection was refused by other side: 10061: No connection co…

Bounty: 100

My steps:

  1. Build image docker build . -t scrapy
  2. Run a container docker run -it -p 8050:8050 --rm scrapy
  3. In container run scrapy project: scrapy crawl foobar -o allobjects.json

This works locally, but on my production server I get error:

[scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 1 times): Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..

Note: I’m NOT using Docker Desktop, neither can I on this server.

Dockerfile

FROM mcr.microsoft.com/windows/servercore:ltsc2019

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

RUN setx /M PATH $('C:UsersContainerAdministratorminiconda3Librarybin;C:UsersContainerAdministratorminiconda3Scripts;C:UsersContainerAdministratorminiconda3;' + $Env:PATH)
RUN Invoke-WebRequest "https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Windows-x86_64.exe" -OutFile miniconda3.exe -UseBasicParsing; 
    Start-Process -FilePath 'miniconda3.exe' -Wait -ArgumentList '/S', '/D=C:UsersContainerAdministratorminiconda3'; 
    Remove-Item .miniconda3.exe; 
    conda install -y -c conda-forge scrapy;

RUN pip install scrapy-splash
RUN pip install scrapy-user-agents
    
#creates root directory if not exists, then enters it
WORKDIR /root/scrapy

COPY scrapy /root/scrapy

settings.py

SPLASH_URL = 'http://localhost:8050/'

OUTPUT with command scrapy crawl foobar -o allobjects.json

2021-09-15 20:12:16 [scrapy.core.engine] INFO: Spider opened
2021-09-15 20:12:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min
)
2021-09-15 20:12:16 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-09-15 20:12:16 [py.warnings] WARNING: C:UsersContainerAdministratorminiconda3libsite-packagesscrapy_splashre
quest.py:41: ScrapyDeprecationWarning: Call to deprecated function to_native_str. Use to_unicode instead.
  url = to_native_str(url)

2021-09-15 20:12:16 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36
2021-09-15 20:12:16 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36
2021-09-15 20:12:17 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 1 times): Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2021-09-15 20:12:17 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) App
leWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
2021-09-15 20:12:18 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 2 times): Connection was refused by other side: 10061: No connection
could be made because the target machine actively refused it..
2021-09-15 20:12:18 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64
) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36
2021-09-15 20:12:19 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 3 times): Connection was refused by other side: 10061: No con
nection could be made because the target machine actively refused it..
2021-09-15 20:12:19 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.example.com via http://localhost:8050/execute>
Traceback (most recent call last):
  File "C:UsersContainerAdministratorminiconda3libsite-packagesscrapycoredownloadermiddleware.py", line 45, in
process_request
    return (yield download_func(request=request, spider=spider))
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 10061: No connection could be made
because the target machine actively refused it..
2021-09-15 20:12:19 [scrapy.core.engine] INFO: Closing spider (finished)
2021-09-15 20:12:19 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 3,
 'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 3,
 'downloader/request_bytes': 4632,
 'downloader/request_count': 3,
 'downloader/request_method_count/POST': 3,
 'elapsed_time_seconds': 3.310168,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2021, 9, 15, 18, 12, 19, 605641),
 'log_count/DEBUG': 6,
 'log_count/ERROR': 2,
 'log_count/INFO': 10,
 'log_count/WARNING': 46,
 'retry/count': 2,
 'retry/max_reached': 1,
 'retry/reason_count/twisted.internet.error.ConnectionRefusedError': 2,
 'scheduler/dequeued': 4,
 'scheduler/dequeued/memory': 4,
 'scheduler/enqueued': 4,
 'scheduler/enqueued/memory': 4,
 'splash/execute/request_count': 1,
 'start_time': datetime.datetime(2021, 9, 15, 18, 12, 16, 295473)}
2021-09-15 20:12:19 [scrapy.core.engine] INFO: Spider closed (finished)

What am I missing?

I already checked here:

UPDATE 1

I included EXPOSE 8050 in my Dockerfile, but get the same error. I tried netstat -a inside the docker container, but 8050 seems not to be in there?

C:rootscrapy>netstat -a

Active Connections

  Proto  Local Address          Foreign Address        State
  TCP    0.0.0.0:135            c60d48724046:0         LISTENING
  TCP    0.0.0.0:5985           c60d48724046:0         LISTENING
  TCP    0.0.0.0:47001          c60d48724046:0         LISTENING
  TCP    0.0.0.0:49152          c60d48724046:0         LISTENING
  TCP    0.0.0.0:49153          c60d48724046:0         LISTENING
  TCP    0.0.0.0:49154          c60d48724046:0         LISTENING
  TCP    0.0.0.0:49155          c60d48724046:0         LISTENING
  TCP    0.0.0.0:49159          c60d48724046:0         LISTENING
  TCP    [::]:135               c60d48724046:0         LISTENING
  TCP    [::]:5985              c60d48724046:0         LISTENING
  TCP    [::]:47001             c60d48724046:0         LISTENING
  TCP    [::]:49152             c60d48724046:0         LISTENING
  TCP    [::]:49153             c60d48724046:0         LISTENING
  TCP    [::]:49154             c60d48724046:0         LISTENING
  TCP    [::]:49155             c60d48724046:0         LISTENING
  TCP    [::]:49159             c60d48724046:0         LISTENING
  UDP    0.0.0.0:5353           *:*
  UDP    0.0.0.0:5355           *:*
  UDP    127.0.0.1:51352        *:*
  UDP    [::]:5353              *:*
  UDP    [::]:5355              *:*


Get this bounty!!!

#StackBounty: #windows #docker #containers #windows-server-2019 Docker build command creates large amount of folders in windowsfilter f…

Bounty: 50

I’m building a new image like so docker build . -t test and copy contents from host OS folder into it:

Dockerfile

FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

RUN mkdir root
RUN cd root
WORKDIR /root
RUN mkdir test
COPY test /root/test
    #USING 'COPY' here instead of 'ADD' because of https://stackoverflow.com/questions/24958140/what-is-the-difference-between-the-copy-and-add-commands-in-a-dockerfile

I get this output in Powershell:

PS D:Programs> docker build . -t test
Sending build context to Docker daemon  1.644GB
Step 1/7 : FROM mcr.microsoft.com/windows/servercore:ltsc2019
ltsc2019: Pulling from windows/servercore
4612f6d0b889: Pull complete
c67ded6868b6: Pull complete
Digest: sha256:1be9c8378b8c32d31538253a4b96169f5138a5f6d4c0a04e8d8f9a80b9ac8097
Status: Downloaded newer image for mcr.microsoft.com/windows/servercore:ltsc2019
 ---> d1724c2d9a84
Step 2/7 : SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
 ---> Running in 1bfefefbe433
Removing intermediate container 1bfefefbe433
 ---> 37de702deb33
Step 3/7 : RUN mkdir root
 ---> Running in e26d6b49ced7


    Directory: C:


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         9/8/2021   1:07 PM                root


Removing intermediate container e26d6b49ced7
 ---> 451c4d3f3ea1
Step 4/7 : RUN cd root
 ---> Running in 74a228f8118f
Removing intermediate container 74a228f8118f
 ---> 3f175ac67f1d
Step 5/7 : WORKDIR /root
 ---> Running in 5f783d5b2332
Removing intermediate container 5f783d5b2332
 ---> 68b24e033f87
Step 6/7 : RUN mkdir test
 ---> Running in 5771bb7a593a


    Directory: C:root


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         9/8/2021   1:08 PM                test


Removing intermediate container 5771bb7a593a
 ---> 35fa0b2af157
Step 7/7 : COPY test /root/test
 ---> 60eab8242865
Successfully built 60eab8242865
Successfully tagged test:latest

It creates EIGHT(!) folders on the host OS in folder "dockerwindowsfilter"

This seems excessive, it looks like a folder for each step in my Dockerfile, so what are all these folders for and could I prevent creation/autodelete the unneeded ones to keep everything tidy?

Host OS: Windows Server 2019 standard Version 1809 17763.2114
Docker: version 20.10.4, build 110e091

Please note that I do NOT use the Docker Desktop application, see here how I installed Docker.

Update 1

I updated my Dockerfile based on @Noam’s suggestion like so:

#RUN mkdir root
#RUN cd root
#WORKDIR /root
WORKDIR /root #creates root directory if not exists, then enters it

After this change, 6 folders (instead of 8) are created in windowsfilter, so my hunch that Docker creates a folder for each command in Dockerfile seems to be correct.


Get this bounty!!!

#StackBounty: #windows-server-2019 #storage-spaces Error attaching or detaching virtual disk

Bounty: 50

I just physically moved my home lab from one house to another and my storage space for my primary array will not re-attach. The message I get is "Error attaching or detaching virtual disk: Failed to attach virtual disk to MyServer."

I tried a few things including Get-VirtualDisk | Where-Object IsManualAttach -eq $true | Set-VirtualDisk -IsManualAttach $false and after multiple attempts I was able to set the value to $false.

I was able to pull these error messages from Event Viewer:

An error occurred during method execution.

Class:      SPACES_VirtualDisk
Method:     Attach
ObjectId:   [REDACTED]
Error Code: 4

And

An error has occurred during method execution.

Class:      MSFT_VirtualDisk
Method:     Attach
Error Code: 4
Error Message:

Failed
Activity ID: {68137d67-7406-4e9b-b566-050f30b9ea14}

Running Get-VirtualDisk | Connect-VirtualDisk in PowerShell returns:

Connect-VirtualDisk : Failed
Activity ID: {90b49a62-37c6-4dc1-aabc-1489763f0696}
At line:1 char:19
+ Get-VirtualDisk | Connect-VirtualDisk
+                   ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (StorageWMI:ROOT/Microsoft/...SFT_VirtualDisk) [Connect-VirtualDisk], CimException
    + FullyQualifiedErrorId : StorageWMI 4,Connect-VirtualDisk

At this point I am not sure what to do. I do have some commercial recovery software from RunTime Software but I would rather avoid doing that as that would require my to buy additional drives to offload all my data onto it. I do have a backup of 85% of it online but I would rather not go that route if possible.


Get this bounty!!!

#StackBounty: #powershell #amazon-ec2 #windows-server #windows-server-2019 Given a virtual disk, get the pool it belongs to

Bounty: 100

I have this monstrosity:

Get-Volume | ForEach-Object {
    $VolObj = $_
    $ParObj = Get-Partition | Where-Object { $_.AccessPaths -contains $VolObj.Path }
    $DiskObj = Get-Disk | Where-Object { $_.Number -eq $ParObj.DiskNumber }
    $PsDriveObj = Get-PSDrive -PSProvider FileSystem | Where-Object { $_.Name -contains $VolObj.DriveLetter }

    Get-StoragePool -IsPrimordial $false | ForEach-Object {
        $pool = $_
        Get-PhysicalDisk -StoragePool $pool | ForEach-Object {
            $disk = $_
            if ($disk.UniqueId -eq $volume_id) {
                $pool_id = $pool.UniqueId 
            }
        }
    }

    ...

}

The problem with this piece of code is in the line if ($disk.uniqueid -eq $volume_id).

Instead of $volume_id I’ve tried .SerialNumber and UniqueId of the multiple objects (VolObj, ParObj, DiskObj and PsDriveObj) but these are either empty or return the wrong the ID (the IDs of these objects are either vol*** since I run it on an EC2 instance or in the format of {...}), so the if is never true.

Is there a straightforward way of achieving this?


Get this bounty!!!

#StackBounty: #windows-server-2019 #smartcard Making (virtual) smartcard available to system service

Bounty: 100

I have a VM running Windows Server 2019 Datacenter Core, which is running a Jenkins build agent as a noninteractive service with its own local user account (.build).

Now I’d like to add code signing. To have at least a semblance of security, I want to use a virtual smartcard that is based on crypto operations on the host, so an attacker who is interested in copying key material would need to break out of the VM.

I have configured a virtual USB CCID reader with a card permanently inserted, and it shows up in Windows:

PS> Get-PnpDevice
...
Error      SmartCardReader Microsoft Usbccid Smartcard Reader (WUDF)                                        USBVID_08E6...
...

First, the Error state is concerning. I have applied this fix, which seems to have improved things a bit (the card shows up OK right after boot), but that doesn’t seem permanent.

The problem where I’m truly stuck however is the security model for smartcard access. Most documentation I could find concerns using the smartcard for logon, for which the policy is simple: the logon UI has access to the card to verify credentials, then passes on this access to the user session.

In my case however, I have a noninteractive logon that isn’t associated with a desktop. When I manually start the SCardSvr and ScDeviceEnum services, I cannot access the card even as Administrator when logged in via SSH:

The Microsoft Smart Card Resource Manager is not running.
SCardAccessStartedEvent: Service is in an unknown state.
CertUtil: -SCInfo command FAILED: 0x80070005 (WIN32: 5 ERROR_ACCESS_DENIED)
CertUtil: Access is denied.

which makes sense, and from the local console, as Administrator, I get

PS> certutil -scinfo
The Microsoft Smart Card Resource Manager is running.
Current reader/card status:
SCardEstablishContext: The Smart Card Resource Manager is not running. 0x8010001d (-2146435043 SCARD_E_NO_SERVICE)
SCardEstablishContext failed for user scope.

The two services also terminate by themselves after two minutes.

Questions:

  • How can I verify that the emulated smartcard is seen as inserted and possibly list the contained certificates?
  • How can I give permission to a user to access the smartcard (certutil -scinfo would probably show that)?
  • How can I import a certificate that already exists on the smartcard into the local user’s key store? Do I need the original P7 certificate, or can I pull that from the card itself?
  • Do I need to do anything to keep the service running continuously, or will resolving the access problems also allow it to demand-start?


Get this bounty!!!

#StackBounty: #hard-drive #windows-server-2019 #failovercluster Windows Disk Event 11: 'The driver detected a controller error on …

Bounty: 150

I’m using several Windows Server 2019 Clusters (e. g. Hyper-V, File Server). On all machines that have clustered roles, I get the following errors (with different harddisk numbers):

Log Name: System
Source:   Disk
Event ID: 11
Level:    Error
Message:  The driver detected a controller error on DeviceHarddisk1DR1.

From my observations, I can conclude, that the error is always thrown on harddisks that are currently offline on one cluster member, because they are online on another cluster member. So it happens on disks that are used by cluster roles for data and disk witness in quorum.

I’m not sure, if that’s just ok in this case and that I can ignore those errors or if there is some misconfiguration and something has to be fixed.

Can someone confirm, that this is normal behaviour or that something might be broken?


Get this bounty!!!

#StackBounty: #hard-drive #windows-server-2019 #failovercluster Windows Disk Event 11: 'The driver detected a controller error on …

Bounty: 150

I’m using several Windows Server 2019 Clusters (e. g. Hyper-V, File Server). On all machines that have clustered roles, I get the following errors (with different harddisk numbers):

Log Name: System
Source:   Disk
Event ID: 11
Level:    Error
Message:  The driver detected a controller error on DeviceHarddisk1DR1.

From my observations, I can conclude, that the error is always thrown on harddisks that are currently offline on one cluster member, because they are online on another cluster member. So it happens on disks that are used by cluster roles for data and disk witness in quorum.

I’m not sure, if that’s just ok in this case and that I can ignore those errors or if there is some misconfiguration and something has to be fixed.

Can someone confirm, that this is normal behaviour or that something might be broken?


Get this bounty!!!

#StackBounty: #hard-drive #windows-server-2019 #failovercluster Windows Disk Event 11: 'The driver detected a controller error on …

Bounty: 150

I’m using several Windows Server 2019 Clusters (e. g. Hyper-V, File Server). On all machines that have clustered roles, I get the following errors (with different harddisk numbers):

Log Name: System
Source:   Disk
Event ID: 11
Level:    Error
Message:  The driver detected a controller error on DeviceHarddisk1DR1.

From my observations, I can conclude, that the error is always thrown on harddisks that are currently offline on one cluster member, because they are online on another cluster member. So it happens on disks that are used by cluster roles for data and disk witness in quorum.

I’m not sure, if that’s just ok in this case and that I can ignore those errors or if there is some misconfiguration and something has to be fixed.

Can someone confirm, that this is normal behaviour or that something might be broken?


Get this bounty!!!

#StackBounty: #hard-drive #windows-server-2019 #failovercluster Windows Disk Event 11: 'The driver detected a controller error on …

Bounty: 150

I’m using several Windows Server 2019 Clusters (e. g. Hyper-V, File Server). On all machines that have clustered roles, I get the following errors (with different harddisk numbers):

Log Name: System
Source:   Disk
Event ID: 11
Level:    Error
Message:  The driver detected a controller error on DeviceHarddisk1DR1.

From my observations, I can conclude, that the error is always thrown on harddisks that are currently offline on one cluster member, because they are online on another cluster member. So it happens on disks that are used by cluster roles for data and disk witness in quorum.

I’m not sure, if that’s just ok in this case and that I can ignore those errors or if there is some misconfiguration and something has to be fixed.

Can someone confirm, that this is normal behaviour or that something might be broken?


Get this bounty!!!

#StackBounty: #hard-drive #windows-server-2019 #failovercluster Windows Disk Event 11: 'The driver detected a controller error on …

Bounty: 150

I’m using several Windows Server 2019 Clusters (e. g. Hyper-V, File Server). On all machines that have clustered roles, I get the following errors (with different harddisk numbers):

Log Name: System
Source:   Disk
Event ID: 11
Level:    Error
Message:  The driver detected a controller error on DeviceHarddisk1DR1.

From my observations, I can conclude, that the error is always thrown on harddisks that are currently offline on one cluster member, because they are online on another cluster member. So it happens on disks that are used by cluster roles for data and disk witness in quorum.

I’m not sure, if that’s just ok in this case and that I can ignore those errors or if there is some misconfiguration and something has to be fixed.

Can someone confirm, that this is normal behaviour or that something might be broken?


Get this bounty!!!