#StackBounty: #python-3.x #continuous-integration #github-actions #miniconda For changes to take effect, close and re-open your current…

Bounty: 100

I have written a github workflow file. I want to run a python program in github actions to validate few changes. I have one environment.yml file which contains all conda environment dependencies required by this program. The thing is, actual program is not running at all, and my workflow is completed with success.

Following is jobs section of workflow.yml file

jobs:
  build-linux:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          ref: refs/pull/${{ github.event.pull_request.number }}/merge

      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: 3.8

      - name: Cache conda
        uses: actions/cache@v2
        env:
          # Increase this value to reset cache if etc/example-environment.yml has not changed
          CACHE_NUMBER: 0
        with:
          path: ~/conda_pkgs_dir
          key:
            ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{hashFiles('**/environment.yml') }}

      - uses: conda-incubator/setup-miniconda@v2
        with:
          activate-environment: test-env
          environment-file: environment.yml
          use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!

      - name: Test
        run: |
          export PATH="./:$PATH"
          conda init bash
          exec bash
          conda activate test-env
          echo "Conda prefix: $CONDA_PREFIX"
          python test.py
        shell: bash

I also tried removing shell:bash in the last step, but this is also giving me the same result.
The logs in last step looks like this:

Run export PATH="./:$PATH"
  export PATH="./:$PATH"
  conda init bash
  exec bash
  conda activate test-env
  echo "Conda prefix: $CONDA_PREFIX"
  python test.py
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.11/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.11/x64/lib
    CONDA_PKGS_DIR: /home/runner/conda_pkgs_dir
no change     /usr/share/miniconda/condabin/conda
no change     /usr/share/miniconda/bin/conda
no change     /usr/share/miniconda/bin/conda-env
no change     /usr/share/miniconda/bin/activate
no change     /usr/share/miniconda/bin/deactivate
no change     /usr/share/miniconda/etc/profile.d/conda.sh
no change     /usr/share/miniconda/etc/fish/conf.d/conda.fish
no change     /usr/share/miniconda/shell/condabin/Conda.psm1
no change     /usr/share/miniconda/shell/condabin/conda-hook.ps1
no change     /usr/share/miniconda/lib/python3.9/site-packages/xontrib/conda.xsh
no change     /usr/share/miniconda/etc/profile.d/conda.csh
modified      /home/runner/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

As we can clearly see, the line echo "Conda prefix: $CONDA_PREFIX" is not getting executed at all, and the workflow terminates with success. We should expect it to either run or fail the job, but nothing happens. The workflow simply ignores these commands and marks the workflow as success.


Get this bounty!!!

#StackBounty: #python #c #python-3.x #ssh #paramiko Is there a method of executing a command then continue the program without command …

Bounty: 100

Hi I am trying to create a function which remotely executes my packet sniffing script on my raspberry pi using paramiko and ssh.

def startPacketReceiver():
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(AutoAddPolicy())
    ssh.connect(RECV_IP_ADDRESS, username="pi", password="raspberry")
    ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command("sudo gcc Code/test.c && sudo ./a.out")
    print("Done")

The test.c file is the packet sniffing script. It will only terminate with a CTRL-C (or equivalent method). It does not terminate naturally/eventually.

I want to be able to start the receiver and then quit the receiver e.g:

startPacketReceiver()
...
stopPacketReceiver()

Currently when I run the python script I never get the "Done" print message, meaning that the program is hung on the exec_command and will not continue until it is terminated.

Additional Info

The test.c file loops infinitely, essentially:

while(1)
{
    saddr_size = sizeof saddr;
    //Receive a packet
    data_size = recvfrom(sock_raw , buffer , 65536 , 0 , &saddr , (socklen_t*)&saddr_size);
    //fprintf(stderr,"%d",data_size);
    if(data_size <0 )
    {
        fprintf(stderr,"Failed to get packetn");
        printf("Recvfrom error , failed to get packetsn");
        return 1;
    }
    //Now process the packet
    ProcessPacket(buffer , data_size);
}

and so to stop it you must CTRL-C it.


Get this bounty!!!

#StackBounty: #python-3.x #pandas #dataframe #merge #pandas-groupby Why does merging unequally matched rows not work on local dataset?

Bounty: 50

I have a pandas dataframe with questions (type = 1) and answers (type = 2). col section_id and type are integer. all other col are string. I want to merge the "answer rows" with their corresponding "question rows" (equal values in section_id) before appending some of the answer rows’ values as extra columns (Ans, ans_t) to their corresponding "question rows".

c = ['pos', 'Ans', 'Q_ID', 'leg', 'que_l', 'ans_l', 'par', 'ans_f', 'que_date', 'ask', 'M_ID', 'part', 'area', 'que_t', 'ans_t', 'ISO', 'con', 'id', 'section_id', 'type', 'dep', 'off']
d = [[None, None, '16-17/1/2017-11-15/1', '16-17', '14.0', None, 'aaa', 'hhh', '2016-11-20', 'Peter Muller', '41749', 'bbb', 'ccc', 'ddd', None, 'eee', 'fff', '111865.q2', 24339851, 1, None, None],
     [None, None, '16-17/24/17-11-09/1', '16-17', '28.0', None, 'aaa', 'hhh', '2016-11-20', 'Peter Muller', '41749', 'bbb', 'ccc', 'ppp', None, 'eee', 'fff', '111867.q1', 24339851, 1, None, None],
     [None, None, '16-17/73/17-10-09/1', '16-17', '69.0', None, 'aaa', 'hhh', '2016-11-20', 'Peter Muller', '41749', 'bbb', 'ccc', 'lll', None, 'eee', 'fff', '111863.r0', 24339851, 1, None, None],
     ['erg', 'wer', '16-17/42/16-10-09/1', '16-17', None, 67.0, 'aaa', 'hhh', '2016-11-20', None, '46753', 'bbb', 'ccc', None, 'ttt', 'eee', 'asd', '111863.r0', 24339851, 2, None, None],
     [None, None, '16-17/12/16-12-08/1', '16-17', '37.0', None, 'aaa', 'hhh', '2016-10-10', 'Peter Muller', '41749', 'bbb', 'qqq', 'rrr', None, 'eee', 'fff', '108143.r0', 24303320, 1, None, None],
     ['erg', 'wer', '16-17/12/16-12-07/1', '16-17', None, 64.0, 'aaa', 'hhh', '2016-10-10', None, '46753', 'bbb', 'qqq', None, 'uuu', 'eee', 'asd', '108143.r0', 24303320, 2, None, None],
     [None, None, '16-17/77/16-12-04/1', '16-17', '46.0', None, 'aaa', 'hhh', '2016-10-08', 'Markus John', '34567', 'ztr', 'yyy', 'nnn', None, 'eee', 'www', '127193.q0', 10343145, 1, None, None],
     ['qwe', 'wer', '16-17/37/17-11-07/1', '16-17', None, 60.0, 'aaa', 'hhh', '2016-12-12', None, '19745', 'bbb', 'gtt', None, 'ooo', 'eee', 'asd', '906213.r0', 23222978, 2, None, None]]
data = pd.DataFrame(d,columns=c)
data.loc[data['type'] == 2, 'Ans.1'] = data['Ans']
data.loc[data['type'] == 2, 'ans_t.1'] = data['ans_t']
my_cols = ['que_t','ans_t','Ans','ans_t','Ans.1','ans_t.1']
data[my_cols] = data.sort_values(['section_id','type']).groupby('section_id')[my_cols].transform(lambda x: x.bfill())
data.dropna(subset=['que_t'],inplace=True)
data.reset_index(drop=True,inplace=True)
print(data)

The code works fine on the minimal reproducible example. Unfortunately the dataset is too large to account for every detail, which is why this example may not necessarily be representative.

Problem: When i run the code on the actual dataset, nothing gets merged, even though i manually checked for section_id duplicates

Before executing the code, i remove empty cells from the dataset

data.where(pd.notnull(data), None)
data.replace(r'^s+$', np.nan, regex=True, inplace=True)

which doesent solve the problem

Question: How do i need to adjust my code in order to account for details (e.g. encoding, formats, ..) in the dataset that could cause it not to merge?

Appendix:

Someone told me to remove data from the dataset gradually, checking each time that the testcase is still reproducible. If some removal results in the testcase not working then reinstate it and remove something else instead. When there’s absolutely nothing that can be removed, you have your minimal data set.

Someone else said i should apply a parsing function to parse the data

def parse(x):
try:
    return int(x)
except ValueError:
    return np.nan
data['col_name'] = data['col_name'].apply(parse)
data.dtypes

Or should I search for non-number string and replace when with NaN ?

replaced_with_nan = data['col_name'].replace(re.compile('D+'), np.nan)
data['col_name'] = replaced_with_nan.astype(np.float)

Here is another approach which as also the answer from Andrej Kesely returns an empty dataframe when used on the actual dataframe

df1 = data.loc[df.type == 1].copy()
df2 = data.loc[df.type == 2].copy()
merged_df = pd.merge(df1, df2, on='section_id', how='outer')
merged_df = merged_df.loc[:,['section_id','que_t_x','ans_t_y','Ans_x','Ans_y']]
merged_df.rename(columns={'que_t_x':'que_t','ans_t_y':'ans_t','Ans_x':'Ans','Ans_y':'Ans.1'}, inplace=True)


Get this bounty!!!

#StackBounty: #python-3.x #telegram-bot #telethon How to forward a message from telegram channel to user using bot

Bounty: 50

I am using python telethon module. I want to send some messages having media files from a public telegram channel to a user using bot. I get the required message by GetMessagesRequest function by logging in as a telegram client i.e. not as bot and channel info using get_entityas bot.

channelInfo=await BOT.get_entity(tl.types.PeerChannel(channelId))
MSG = await CLIENT(functions.channels.GetMessagesRequest( channel=channelInfo.username, id=[MessageId]))
Forward_MSG = MSG.messages[0]

Using await BOT.send_message(USER, MSG) or await BOT.send_file(USER, MSG) gives telethon.errors.rpcerrorlist.MediaEmptyError: The provided media object is invalid (caused by SendMediaRequest)

Using await BOT.forward_messages(USER, MSG) gives TypeError: Cannot cast NoneType to any kind of Peer.

How can I send Forward_MSG to a user as a bot ?


Get this bounty!!!

#StackBounty: #python #python-3.x #multithreading #python-requests Python multi connection downloader resuming after pausing makes down…

Bounty: 100

I have written a Python script that downloads a single file using 32 connections if available.

I have written a multiconnection downloader that works fine without pausing, but won’t stop downloading after resuming, the progress would go beyond 100%…

Like this:

Download mode: Multi-thread (press Space to pause/resume, press Escape to stop)                                                                                                             
[████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████] 120% completed, paused: False                                    
Download mode: Multi-thread (press Space to pause/resume, press Escape to stop)                                                                                                             
1798.08 MiB downloaded, 1489.83 MiB total, -308.25 MiB remaining, download speed: 22.73 MiB/s                                                                                               
Minimum speed: 0.00 MiB/s, average speed: 4.54 MiB/s, maximum speed: 75.00 MiB/s                                                                                                            
Task started on 2021-08-09 16:57:03, 00:06:35 elapsed, ETA: -1:59:47

After progress exceeds 100%, there will be error messages like this:

Exception in thread Thread-78:
Traceback (most recent call last):
  File "C:Program FilesPython39libthreading.py", line 973, in _bootstrap_inner
    self.run()
  File "C:Program FilesPython39libthreading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "D:MyScriptdownloader.py", line 70, in multidown
    mm[position: position+len(chunk)] = chunk
IndexError: mmap slice assignment is wrong size

(The above doesn’t include all of the error message)

I have encountered all sorts of errors after resuming, but most importantly, the server will often send extra bytes from previous request, whose connection is dead and needless to say this breaks the whole code.

How should I implement pause and resume correctly?

I am thinking about multiprocessing, I assume the sessions and connections are all PID and port number related, and so far I haven’t encountered a new run of the script that received extra bytes from previous runs of the script, so I guess using another process with a new PID and new port number plus requests.session() plus {'connection': 'close'} for each download should guarantee that no extra bytes from previous connections will be received, I just don’t know how to share variables between processes…

The code:
downloader.py

import json
import keyboard
import os
import re
import requests
import sys
import time
import validators
from collections import deque
from datetime import datetime, timedelta
from math import inf
from mmap import mmap
from pathlib import Path
from ping3 import ping
from reprint import output
from threading import Thread


def timestring(sec):
    sec = int(sec)
    m, s = divmod(sec, 60)
    h, m = divmod(m, 60)
    return f'{h:02d}:{m:02d}:{s:02d}'


class Downloader:
    def __init__(self):
        self.recent = deque([0] * 12, maxlen=12)
        self.recentspeeds = deque([0] * 200, maxlen=200)
        self.paused = False
        self.progress = dict()

    class Multidown:
        def __init__(self, obj, id):
            self.count = 0
            self.position = 0
            self.completed = False
            self.id = id
            self.parent = obj
        
        def multidown(self, url, start, end):
            interrupted = False
            s = requests.session()
            s.headers.update({'connection': 'close', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'})
            r = s.get(
                url, headers={'range': 'bytes={0}-{1}'.format(start, end)}, stream=True)
            length = int(r.headers['content-length'])
            while end - length + (self.id != self.parent.progress['connections'] - 1) != start or r.status_code != 206:
                r.close()
                s.close()
                del r
                del s
                time.sleep(0.02)
                s = requests.session()
                r = s.get(
                    url, headers={'range': 'bytes={0}-{1}'.format(start, end)}, stream=True)
                length = int(r.headers['content-length'])
            self.position = start
            
            for chunk in r.iter_content(1048576):
                if self.parent.paused:
                    self.parent.mm.flush()
                    r.connection.close()
                    r.close()
                    s.close()
                    del r
                    del s
                    interrupted = True
                    break
                if chunk:
                    self.parent.mm[self.position: self.position+len(chunk)] = chunk
                    self.count += len(chunk)
                    self.position += len(chunk)
                    self.parent.progress[self.id]['count'] = self.count
                    self.parent.progress[self.id]['position'] = self.position
            if not interrupted:
                r.close()
                s.close()
            if self.count == self.parent.progress[self.id]['length']:
                self.completed = True
                self.parent.progress[self.id]['completed'] = True
                self.parent.mm.flush()
    
    class Singledown:
        def __init__(self):
            self.count = 0
        def singledown(self, url, path):
            with requests.get(url, stream=True) as r:
                with path.open('wb') as file:
                    for chunk in r.iter_content(1048576):
                        if chunk:
                            self.count += len(chunk)
                            file.write(chunk)

    def download(self, url, filepath, num_connections=32, overwrite=False):
        singlethread = False
        threads = []
        bcontinue = False
        filepath = filepath.replace('\', '/')
        if (not re.match('^[a-zA-Z]:/(((?![<>:"/|?*]).)+((?<![ .])/)?)*$', filepath) or
                not Path(filepath[:3]).exists()):
            print('Invalid windows file path has been inputted, process will now stop.')
            return
        if not validators.url(url):
            print('Invalid url been inputted, process will now stop.')
            return
        if url.lower().startswith('ftp://'):
            print(
                "`requests` module doesn't suport File Transfer Protocol, process will now stop")
            return
        path = Path(filepath)
        if not path.exists():
            bcontinue = True
        else:
            if path.is_file():
                if overwrite:
                    bcontinue = True
                else:
                    while True:
                        answer = input(
                            f'`{filepath}` already exists, do you want to overwrite it? n(Yes, No):').lower()
                        if answer in ['y', 'yes', 'n', 'no']:
                            if answer.startswith('y'):
                                os.remove(filepath)
                                bcontinue = True
                            break
                        else:
                            print('Invalid input detected, retaking input.')
        if not bcontinue:
            print(
                f'Overwritting {filepath} has been aborted, process will now stop.')
            return
        bcontinue = False
        server = url.split('/')[2]
        ok = ping(server, timeout=2)
        if ok == False:
            print(
                'The server of the inputted url is non-existent, process will now stop.')
            return
        if ok:
            bcontinue = True
        if not ok:
            print('Connection has timed out, will reattempt to ping server 5 times.')
            for i in range(5):
                print(
                    f'Reattempting to ping server, retrying {i + 1} out of 5')
                ok = ping(server, timeout=2)
                if ok:
                    print(
                        f'Connection successful on retry {i + 1}, process will now continue.')
                    bcontinue = True
                    break
                else:
                    print(f'Retry {i + 1} out of 5 timed out' + (i != 4)
                          * ', reattempting in 1 second.' + (i == 4) * '.')
                    time.sleep(1)
        if not bcontinue:
            print('Failed to connect server, connection timed out, process will now stop')
            return
        bcontinue = False
        head = requests.head(url)
        if head.status_code == 200:
            bcontinue = True
        else:
            for i in range(5):
                print(f'Server responce is invalid, retrying {i + 1} out of 5')
                head = requests.head(url)
                if head.status_code == 200:
                    print(
                        f'Connection successful on retry {i + 1}, process will now continue.')
                    bcontinue = True
                    break
                else:
                    print(f'Retry {i + 1} out of 5 failed to access data' +
                          (i != 4) * ', reattempting in 1 second.' + (i == 4) * '.')
                    time.sleep(1)
        if not bcontinue:
            print("Can't establish a connection with access to data, can't download target file, process will now stop.")
            return
        folder = '/'.join(filepath.split('/')[:-1])
        Path(folder).mkdir(parents=True, exist_ok=True)
        headers = head.headers
        total = headers.get('content-length')
        if not total:
            print(
                f'Cannot find the total length of the content of {url}, the file will be downloaded using a single thread.')
            started = datetime.now()
            print('Task started on %s.' %
                    started.strftime('%Y-%m-%d %H:%M:%S'))
            sd = self.Singledown()
            th = Thread(target=sd.singledown, args=(url, path))
            threads.append(sd)
            th.start()
            total = inf
            singlethread = True
        else:
            total = int(total)
            if not headers.get('accept-ranges'):
                print(
                    'Server does not support the `range` parameter, the file will be downloaded using a single thread.')
                started = datetime.now()
                print('Task started on %s.' %
                        started.strftime('%Y-%m-%d %H:%M:%S'))
                sd = self.Singledown()
                th = Thread(target=sd.singledown, args=(url, path))
                threads.append(sd)
                th.start()
                singlethread = True
            else:
                segment = total / num_connections
                started = datetime.now()
                lastpressed = started
                path.touch()
                file = path.open('wb')
                file.seek(total - 1)
                file.write(b'')
                file.close()
                file = path.open(mode='r+b')
                self.mm = mmap(file.fileno(), 0)
                print('Task started on %s.' %
                        started.strftime('%Y-%m-%d %H:%M:%S'))
                self.progress['total'] = total
                self.progress['connections'] = num_connections
                for i in range(num_connections):
                    md = self.Multidown(self, i)
                    start = int(segment * i)
                    end = int(segment * (i + 1)) - (i != num_connections - 1)
                    length = end - start + (i != num_connections - 1)
                    th = Thread(target=md.multidown, args=(
                        url, start, end))
                    threads.append(md)
                    self.progress[i] = dict()
                    self.progress[i]['start'] = start
                    self.progress[i]['position'] = start
                    self.progress[i]['end'] = end
                    self.progress[i]['count'] = 0
                    self.progress[i]['length'] = length
                    self.progress[i]['completed'] = False
                    th.start()
                Path(filepath + '.progress.json').write_text(json.dumps(self.progress, indent=4))
        downloaded = 0
        totalMiB = total / 1048576
        speeds = []
        interval = 0.04
        with output(initial_len=5, interval=0) as dynamic_print:
            while True:
                Path(filepath + '.progress.json').write_text(json.dumps(self.progress, indent=4))
                status = sum([i.completed for i in threads])
                downloaded = sum(i.count for i in threads)
                self.recent.append(downloaded)
                done = int(100 * downloaded / total)
                doneMiB = downloaded / 1048576
                gt0 = len([i for i in self.recent if i])
                if not gt0:
                    speed = 0
                else:
                    recent = list(self.recent)[12 - gt0:]
                    if len(recent) == 1:
                        speed = recent[0] / 1048576 / interval
                    else:
                        diff = [b - a for a, b in zip(recent, recent[1:])]
                        speed = sum(diff) / len(diff) / 1048576 / interval
                speeds.append(speed)
                self.recentspeeds.append(speed)
                nzspeeds = [i for i in speeds if i]
                if nzspeeds:
                    minspeed = min(nzspeeds)
                else:
                    minspeed = 0
                maxspeed = max(speeds)
                now = datetime.now()
                elapsed = (now - started).total_seconds()
                meanspeed = downloaded / elapsed / 1048576
                remaining = totalMiB - doneMiB
                dynamic_print[0] = '[{0}{1}] {2}'.format(
                    'u2588' * done, 'u00b7' * (100-done), str(done)) + '% completed' + (not singlethread) * ', paused: {0}'.format(self.paused)
                dynamic_print[1] = 'Download mode: ' + singlethread * 
                    'Single-thread' + (not singlethread) * 'Multi-thread (press Space to pause/resume, press Escape to stop)'
                dynamic_print[2] = '{0:.2f} MiB downloaded, {1:.2f} MiB total, {2:.2f} MiB remaining, download speed: {3:.2f} MiB/s'.format(
                    doneMiB, totalMiB, remaining, speed)
                if speed and total != inf:
                    eta = timestring(remaining / speed)
                else:
                    eta = '99:59:59'
                dynamic_print[3] = 'Minimum speed: {0:.2f} MiB/s, average speed: {1:.2f} MiB/s, maximum speed: {2:.2f} MiB/s'.format(
                    minspeed, meanspeed, maxspeed)
                dynamic_print[4] = 'Task started on {0}, {1} elapsed, ETA: {2}'.format(
                    started.strftime('%Y-%m-%d %H:%M:%S'), timestring(elapsed), eta)
                if keyboard.is_pressed('space'):
                    if not singlethread:
                        pressed = datetime.now()
                        if (pressed - lastpressed).total_seconds() > 0.5:
                            lastpressed = pressed
                            if self.paused:
                                for i, md in enumerate(threads):
                                    if not md.completed:
                                        th = Thread(target=md.multidown, args=(
                                            url, self.progress[i]['position'], self.progress[i]['end']))
                                        th.start()
                            self.paused = not self.paused
                if keyboard.is_pressed('esc'):
                    if not singlethread:
                        ended = datetime.now()
                        self.paused = True
                        break
                if status == len(threads):
                    if not singlethread:
                        self.mm.close()
                    ended = datetime.now()
                    break
                time.sleep(interval)
        time_spent = (ended - started).total_seconds()
        meanspeed = total / time_spent / 1048576
        status = sum([i.completed for i in threads])
        if status == len(threads):
            print('Task completed on {0}, total time elapsed: {1}, average speed: {2:.2f} MiB/s'.format(
                ended.strftime('%Y-%m-%d %H:%M:%S'), timestring(time_spent), meanspeed))
        else:
            print('Task interrupted on {0}, total time elapsed: {1}, average speed: {2:.2f} MiB/s'.format(
                ended.strftime('%Y-%m-%d %H:%M:%S'), timestring(time_spent), meanspeed))

if __name__ == '__main__':
    d = Downloader()
    d.download(*sys.argv[1:])

For testing purposes this is a dumbed-down version of the script, with all checks removed while retaining the same functionality (sorry it really takes all these lines to show the download information):

import json
import os
import requests
import sys
import time
from collections import deque
from datetime import datetime, timedelta
from math import inf
from mmap import mmap
from pathlib import Path
from reprint import output
from threading import Thread


def timestring(sec):
    sec = int(sec)
    m, s = divmod(sec, 60)
    h, m = divmod(m, 60)
    return f'{h:02d}:{m:02d}:{s:02d}'


class Downloader:
    def __init__(self):
        self.recent = deque([0] * 12, maxlen=12)
        self.recentspeeds = deque([0] * 200, maxlen=200)
        self.paused = False
        self.progress = dict()
        self.UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'

    class Multidown:
        def __init__(self, obj, id):
            self.count = 0
            self.position = 0
            self.completed = False
            self.id = id
            self.parent = obj
            self.UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'
        
        def multidown(self, url, start, end):
            interrupted = False
            s = requests.session()
            s.headers.update({'connection': 'close', 'user-agent': self.UA})
            r = s.get(
                url, headers={'range': 'bytes={0}-{1}'.format(start, end)}, stream=True)
            length = int(r.headers['content-length'])
            while end - length + (self.id != self.parent.progress['connections'] - 1) != start or r.status_code != 206:
                r.close()
                s.close()
                del r
                del s
                time.sleep(0.02)
                s = requests.session()
                r = s.get(
                    url, headers={'range': 'bytes={0}-{1}'.format(start, end)}, stream=True)
                length = int(r.headers['content-length'])
            self.position = start
            
            for chunk in r.iter_content(1048576):
                if self.parent.paused:
                    self.parent.mm.flush()
                    r.connection.close()
                    r.close()
                    s.close()
                    del r
                    del s
                    interrupted = True
                    break
                if chunk:
                    self.parent.mm[self.position: self.position+len(chunk)] = chunk
                    self.count += len(chunk)
                    self.position += len(chunk)
                    self.parent.progress[self.id]['count'] = self.count
                    self.parent.progress[self.id]['position'] = self.position
            if not interrupted:
                r.close()
                s.close()
            if self.count == self.parent.progress[self.id]['length']:
                self.completed = True
                self.parent.progress[self.id]['completed'] = True
                self.parent.mm.flush()
    

    def download(self, url, filepath, num_connections=32, overwrite=False):
        singlethread = False
        threads = []
        bcontinue = False
        filepath = filepath.replace('\', '/')
        if Path(filepath).exists():
            os.remove(filepath)
        folder = '/'.join(filepath.split('/')[:-1])
        Path(folder).mkdir(parents=True, exist_ok=True)
        head = requests.head(url, headers={'user-agent': self.UA})
        path = Path(filepath)
        headers = head.headers
        total = headers.get('content-length')
        if total:
            total = int(total)
            if headers.get('accept-ranges'):
                segment = total / num_connections
                started = datetime.now()
                lastpressed = started
                path.touch()
                file = path.open('wb')
                file.seek(total - 1)
                file.write(b'')
                file.close()
                file = path.open(mode='r+b')
                self.mm = mmap(file.fileno(), 0)
                print('Task started on %s.' %
                        started.strftime('%Y-%m-%d %H:%M:%S'))
                self.progress['total'] = total
                self.progress['connections'] = num_connections
                for i in range(num_connections):
                    md = self.Multidown(self, i)
                    start = int(segment * i)
                    end = int(segment * (i + 1)) - (i != num_connections - 1)
                    length = end - start + (i != num_connections - 1)
                    th = Thread(target=md.multidown, args=(
                        url, start, end))
                    threads.append(md)
                    self.progress[i] = dict()
                    self.progress[i]['start'] = start
                    self.progress[i]['position'] = start
                    self.progress[i]['end'] = end
                    self.progress[i]['count'] = 0
                    self.progress[i]['length'] = length
                    self.progress[i]['completed'] = False
                    th.start()
                Path(filepath + '.progress.json').write_text(json.dumps(self.progress, indent=4))
        downloaded = 0
        totalMiB = total / 1048576
        speeds = []
        interval = 0.04
        with output(initial_len=5, interval=0) as dynamic_print:
            while True:
                Path(filepath + '.progress.json').write_text(json.dumps(self.progress, indent=4))
                status = sum([i.completed for i in threads])
                downloaded = sum(i.count for i in threads)
                self.recent.append(downloaded)
                done = int(100 * downloaded / total)
                doneMiB = downloaded / 1048576
                gt0 = len([i for i in self.recent if i])
                if not gt0:
                    speed = 0
                else:
                    recent = list(self.recent)[12 - gt0:]
                    if len(recent) == 1:
                        speed = recent[0] / 1048576 / interval
                    else:
                        diff = [b - a for a, b in zip(recent, recent[1:])]
                        speed = sum(diff) / len(diff) / 1048576 / interval
                speeds.append(speed)
                self.recentspeeds.append(speed)
                nzspeeds = [i for i in speeds if i]
                if nzspeeds:
                    minspeed = min(nzspeeds)
                else:
                    minspeed = 0
                maxspeed = max(speeds)
                now = datetime.now()
                elapsed = (now - started).total_seconds()
                meanspeed = downloaded / elapsed / 1048576
                remaining = totalMiB - doneMiB
                dynamic_print[0] = '[{0}{1}] {2}'.format(
                    'u2588' * done, 'u00b7' * (100-done), str(done)) + '% completed' + (not singlethread) * ', paused: {0}'.format(self.paused)
                dynamic_print[1] = 'Download mode: ' + singlethread * 
                    'Single-thread' + (not singlethread) * 'Multi-thread (press Space to pause/resume, press Escape to stop)'
                dynamic_print[2] = '{0:.2f} MiB downloaded, {1:.2f} MiB total, {2:.2f} MiB remaining, download speed: {3:.2f} MiB/s'.format(
                    doneMiB, totalMiB, remaining, speed)
                if speed and total != inf:
                    eta = timestring(remaining / speed)
                else:
                    eta = '99:59:59'
                dynamic_print[3] = 'Minimum speed: {0:.2f} MiB/s, average speed: {1:.2f} MiB/s, maximum speed: {2:.2f} MiB/s'.format(
                    minspeed, meanspeed, maxspeed)
                dynamic_print[4] = 'Task started on {0}, {1} elapsed, ETA: {2}'.format(
                    started.strftime('%Y-%m-%d %H:%M:%S'), timestring(elapsed), eta)
                if PAUSE:
                    if not singlethread:
                        pressed = datetime.now()
                        if (pressed - lastpressed).total_seconds() > 0.5:
                            lastpressed = pressed
                            if self.paused:
                                for i, md in enumerate(threads):
                                    if not md.completed:
                                        th = Thread(target=md.multidown, args=(
                                            url, self.progress[i]['position'], self.progress[i]['end']))
                                        th.start()
                            self.paused = not self.paused
                if status == len(threads):
                    if not singlethread:
                        self.mm.close()
                    ended = datetime.now()
                    break
                time.sleep(interval)
        time_spent = (ended - started).total_seconds()
        meanspeed = total / time_spent / 1048576
        status = sum([i.completed for i in threads])
        if status == len(threads):
            print('Task completed on {0}, total time elapsed: {1}, average speed: {2:.2f} MiB/s'.format(
                ended.strftime('%Y-%m-%d %H:%M:%S'), timestring(time_spent), meanspeed))
        else:
            print('Task interrupted on {0}, total time elapsed: {1}, average speed: {2:.2f} MiB/s'.format(
                ended.strftime('%Y-%m-%d %H:%M:%S'), timestring(time_spent), meanspeed))

if __name__ == '__main__':
    import hashlib
    global PAUSE
    PAUSE = False
    chash = '5674E59283D95EFE8C88770515A9BBC80CBB77CB67602389FD91DEF26D26AED2'
    d = Downloader()
    if sys.argv[1] == '0':
        d.download('http://ipv4.download.thinkbroadband.com/1GB.zip', 'C:/test/1GB.zip')
    elif sys.argv[1] == '1':
        th1 = Thread(target=d.download, args=('http://ipv4.download.thinkbroadband.com/1GB.zip', 'C:/test/1GB.zip'))
        th1.start()
        def test():
            while th1.is_alive():
                global PAUSE
                PAUSE = not PAUSE
                time.sleep(10)
        th2 = Thread(target=test)
        th2.start()
        while th1.is_alive():
            pass
    sha256_hash = hashlib.sha256()
    with open('C:/test/1GB.zip',"rb") as f:
        for byte_block in iter(lambda: f.read(1048576),b""):
            sha256_hash.update(byte_block)
    print(sha256_hash.hexdigest().lower() == chash.lower())

The url isn’t accessible without a VPN in my locale, and test 0 always results True, that is, if the connection hasn’t gone dead during the download, and test 1 sometimes results True, sometimes results False, sometimes it doesn’t finish(progress bar goes beyond 100%)…

How can my code be salvaged?


Get this bounty!!!

#StackBounty: #python #python-3.x #playing-cards Text Based Blackjack

Bounty: 50

I wanted to make a blackjack game to use while bored, but it turned out to be far more complicated than I thought.

from random import choice, randint

MASTER_DECK = ["A", "A", "A", "A",
               "2", "2", "2", "2",
               "3", "3", "3", "3",
               "4", "4", "4", "4",
               "5", "5", "5", "5",
               "6", "6", "6", "6",
               "7", "7", "7", "7",
               "8", "8", "8", "8",
               "9", "9", "9", "9",
               "10", "10", "10", "10",
               "J", "J", "J", "J",
               "Q", "Q", "Q", "Q",
               "K", "K", "K", "K"]


def setup(deck):
    """Sets up all game variables"""
    # Initialize all of the hands
    player_hand, deck = pick_cards(deck)
    dealer_hand, deck = pick_cards(deck)
    return deck, player_hand, dealer_hand


def pick_cards(deck):
    """Deals two random cards"""
    hand = []
    if len(deck) <= 6:
        deck = MASTER_DECK.copy()
    for card in range(0, 2):
        chosen_card = choice(deck)
        hand.append(chosen_card)
        deck.remove(chosen_card)
    return hand, deck


def print_ui(player_hand, dealer_hand, deck, game_state):
    """Prints out the display that tells the user there cards"""
    print()
    if game_state == "player_dealing":
        print("The dealer has these cards:n_, " + ", ".join(dealer_hand[1:]))
        print()
        print("You have these cards:n" + ", ".join(player_hand))
        print()
        print(f"There are {len(deck)} cards left in the deck")
    elif game_state == "dealer_dealing":
        print("The dealer has these cards:n" + ", ".join(dealer_hand))
        print()
        print("You have these cards:n" + ", ".join(player_hand))
        print()
        if have_won(player_hand, dealer_hand):
            print("You have beaten the dealer.")
        else:
            print("You have not beaten the dealer.")
    else:
        print("Something has gone wrong")
        while True:
            pass


def have_won(player_hand, dealer_hand):
    """Checks if the player has won"""
    numeric_player_hand = numeric_cards(player_hand.copy())
    player_hand_total = 0
    for card in numeric_player_hand:
        player_hand_total += card
    numeric_dealer_hand = numeric_cards(dealer_hand.copy())
    dealer_hand_total = 0
    for card in numeric_dealer_hand:
        dealer_hand_total += card
    if dealer_hand_total > 21:
        if player_hand_total > 21:
            return False
        return True
    if dealer_hand_total == 21:
        return False
    if dealer_hand_total < 21:
        if dealer_hand_total < player_hand_total <= 21:
            return True
        return False


def betting_phase(tokens):
    """Takes the users bet"""
    print(f"You have {tokens} tokens.")
    while True:
        try:
            bet = int(input("Please enter you bet: "))
            if int(bet) > 0:
                if (tokens - bet) >= 0:
                    break
                print("Do not bet more than you have.")
            else:
                print("Please enter a number greater than zero.")
        except ValueError:
            print("Please enter a number.")
    return tokens - bet, bet


def player_dealing(deck, player_hand, game_state):
    """Handles dealing to the player"""
    if not deck:
        print("As there are no more cards left, the round ends.")
        game_state = "dealer_dealing"
    else:
        while True:
            user_command = input("Would you like to hit or to stay? (H/S): ").lower()
            if user_command == "h":
                chosen_card = choice(deck)
                player_hand.append(chosen_card)
                deck.remove(chosen_card)
                break
            elif user_command == "s":
                game_state = "dealer_dealing"
                break
            else:
                print("Please only enter H for hit or S for stay.")
    return deck, player_hand, game_state


def dealer_dealing(deck, dealer_hand):
    """Handles dealing to the dealer"""
    while True:
        if not deck:
            break
        numeric_dealer_hand = numeric_cards(dealer_hand.copy())
        hand_total = 0
        for card in numeric_dealer_hand:
            hand_total += card
        if hand_total < 16:
            chosen_card = choice(deck)
            dealer_hand.append(chosen_card)
            deck.remove(chosen_card)
        elif hand_total == 16:
            if randint(0, 1):
                chosen_card = choice(deck)
                dealer_hand.append(chosen_card)
                deck.remove(chosen_card)
            else:
                break
        elif 11 in numeric_dealer_hand and hand_total > 21:
            for card_number, card in enumerate(numeric_dealer_hand):
                if card == 11:
                    numeric_dealer_hand[card_number] = 1
        else:
            break
    return deck, dealer_hand


def numeric_cards(hand):
    """Turns card letters into their number values"""
    for card_number, card in enumerate(hand):
        if card == "J" or card == "Q" or card == "K":
            hand[card_number] = 10
        elif card == "A":
            hand[card_number] = 11
        else:
            hand[card_number] = int(hand[card_number])
    hand_total = 0
    for card in hand:
        hand_total += card
    if hand_total > 21 and 11 in hand:
        for card_number, card in enumerate(hand):
            if card == 11:
                hand[card_number] = 1
    return hand


def play_again():
    """Allows user to play again or quit"""
    while True:
        play_again = input("Do you want to play again? (Y/N): ").lower()
        if play_again == "y":
            break
        elif play_again == "n":
            quit()
        print("Please only enter a Y or N")


deck = MASTER_DECK.copy()
tokens = 200

while True:
    game_state = "betting"
    playing_game = True

    deck, player_hand, dealer_hand = setup(deck)

    while playing_game:
        if game_state == "betting":
            tokens, bet = betting_phase(tokens)
            game_state = "player_dealing"
        else:
            print_ui(player_hand, dealer_hand, deck, game_state)
            deck, player_hand, game_state = player_dealing(deck, player_hand, game_state)
            if game_state == "dealer_dealing":
                deck, dealer_hand = dealer_dealing(deck, dealer_hand)
                if have_won(player_hand, dealer_hand):
                    tokens += 2 * bet
                print_ui(player_hand, dealer_hand, deck, game_state)
                playing_game = False
    if tokens:
        play_again()
    else:
        input("You have no more tokens to spend. Hit enter to quit.")
        quit()

Originally I wanted to add some AI players as well as including the split and double functions, but the code got so complicated that I thought it would be better not to include those unless the code was cleaned up.

Is there any way to clean this up and make it easier to add more features? As well, is there anything else that could be made better?


Get this bounty!!!

#StackBounty: #python #python-3.x #django #localhost #domain-name How do I access my django app on other devices in my environment usin…

Bounty: 100

I have a django app which I hosted locally on my system, now since other devices are connected and on the same local network as my django app, they too are able to access the website by typing the ipaddress:port of my django app host device.

But I wanted those devices also to access my django app using some domain name, but it’s not working. Here is what I tried.

Edited the hosts file on windows and added

127.0.0.1 anythingyoulike.com 
127.0.0.1 www.anythingyoulike.com
127.0.0.1. blog.anythingyoulike.com 

added our custom domain to the allowed host on our settings.py

ALLOWED_HOSTS = ['www.anythingyoulike.com', 'blog.anythingyoulike.com', 'anythingyoulike.com']

But other devices on my hotspot network are unable to access using these domain names, and only my devices where I hosted my django website is able to access it.

Note : There are Android mobile devices too where I want to access using domain name on my local environment


Get this bounty!!!

#StackBounty: #python #python-3.x Scraping webelements if found

Bounty: 100

Im currently working doing a webscrape which in my case if sydsvenskan but will be a template over time for other sites.

I have currently done:

from typing import Optional

import attr
import requests
from bs4 import BeautifulSoup
from loguru import logger
from requests import RequestException
from requests.exceptions import (
    ConnectionError,
    ConnectTimeout,
    ProxyError,
    ReadTimeout,
    ChunkedEncodingError,
    SSLError,
    Timeout
)


@attr.dataclass
class Data:
    store: str = attr.ib(factory=str)
    name: Optional[str] = attr.ib(factory=str)
    info: Optional[str] = attr.ib(factory=str)
    image: Optional[str] = attr.ib(factory=str)


class Info:
    def from_page(url: str) -> Data:
        while True:
            try:
                with requests.get(url) as response:
                    if response.status_code == 404:
                        return Data(store="Sydsvenskan")
                    if response.ok:
                        doc = BeautifulSoup(response.text, 'html.parser')
                    else:
                        response.raise_for_status()

                name = doc.select_one('span.prose-title')
                info = doc.select_one('div.article__preamble-wrapper')
                image = doc.select_one('img.article-image')

                return Data(
                    store="Sydsvenskan",
                    name=name.text.strip() if name else '',
                    info=info.text.strip() if info else '',
                    image=image['src'] if image else '',
                )
            except (
                    ReadTimeout,
                    Timeout,
                    ConnectTimeout,
                    ConnectionError,
                    ChunkedEncodingError,
                    SSLError,
                    ProxyError
            ) as err:
                logger.info(f"{type(err).__name__} at line {err.__traceback__.tb_lineno} of {__file__}: {err}")
                continue

            except RequestException as err:
                logger.exception(f"{type(err).__name__} at line {err.__traceback__.tb_lineno} of {__file__},: {err}")
                raise RequestException from err

            except Exception as err:
                logger.exception(f"{type(err).__name__} at line {err.__traceback__.tb_lineno} of {__file__},: {err}")
                raise Exception from err


def main():
    info = Info.from_page(
        url="https://www.sydsvenskan.se/2021-07-14/flera-anhallna-for-senaste-tidens-skjutningar-i-malmo")
    if info is None:
        logger.info('No new payload')
    else:
        logger.info(f'New payload: {info}')


if __name__ == '__main__':
    main()

The current "template" I have created is meant to be able to add more news sites in the future and to be able to use this as a template. We do know that each site is different so we will never have any similar but the template should make it easy to adept in the future.

I wonder if there is anything I can improve from here? I am mostly "worried" about the return of Data where I use the if else statement inside the dataclass, not sure if that is a correct way to do it?


Get this bounty!!!

#StackBounty: #python #python-3.x #wxpython wxPython application does not respond

Bounty: 50

I have a complex application, with a GUI that needs to dialogue with some I/O devices and with some WebAPI. I put my wx.Frame class in the main file, as I read that the GUI should be in the main thread to avoid freezing

if __name__ == "__main__":
    app = wx.App()
    frame = Window()
    app.MainLoop()

but still the GUI freezes very often, and sometimes it doesn’t show at all and a message saying "My_app is not responding" appears.
All the I/O and webAPI management is done in separate threads that are created by frame. The only GUI elements that are not in the main file are the pages that compose my notebook

from PageOne import PageOne
from PageTwo import PageTwo
from PageThree import PageThree

...

self.page1 = PageOne(self.nb)
self.page2 = PageTwo(self.nb)
self.page3 = PageThree(self.nb)

self.nb.AddPage(self.page1, "Page1")
self.nb.AddPage(self.page2, "Page2")
self.nb.AddPage(self.page3, "Page3")

All the communications between secondary threads and the GUI are done using wx.lib.newevent.NewEvent() in the main file and wx.PostEvent(self.parent, my_evt) in the threads.

I am using wxpython 4.1.1 and Ubuntu 20.04.2 LTS.

Any suggestion on how to prevent the GUI from not responding or freezing? Is maybe a better idea to use multiprocessing instead of multithreading? I know that threads are usually better for I/O applications…but is it still true in my case where the threads are all enless loops?

def run(self):
    while True:
        do_stuff()


Get this bounty!!!