#StackBounty: #bash #shell #sh #linx Generating a domain and IP blacklist with Bash

Bounty: 50

This is a project to pull domain & IP blacklists from various sources and compile them into one list. There are some whitelists included that are applied when a blacklist is built.

It draws blacklist and whitelist source entries from this JSON document. The "rule" that gets applied is intended to reduce list entries into what is strictly either a domain, IPv4 address, or IPv6 address. Such a format is designated by the "format" field. Each format is placed into its own list, except the "domain" format which is included in all three.

Notes on improvements from any related aspect are welcome and appreciated!

create_builds.sh

#!/usr/bin/env bash
set -euo pipefail

downloads=$(mktemp -d)
trap 'rm -rf "$downloads"' EXIT || exit 1

# params: list name, sort column, cache dir
sort_list() {
    sort -o "$1" -k "$2" -u -S 90% --parallel=4 -T "$3" "$1"
}

for color in 'white' 'black'; do
    cache_dir="${downloads}/${color}"

    jq --arg color "$color" 'to_entries[] | select(.value.color == $color)' sources.json |
        jq -r -s 'from_entries | keys[] as $k | "($k)#(.[$k] | .mirrors)"' |
        while IFS=$'#' read -r key mirrors; do
            echo "$mirrors" | tr -d '[]"' | tr -s ',' "t" | gawk -v key="$key" '{
                if ($0 ~ /.tar.gz$/ || /.zip$/) {
                    printf "%sn out=%s.%sn",$0,key,gensub(/^(.*[/])?[^.]*[.]?/, "", 1, $0)
                } else {
                    printf "%sn out=%s.txtn",$0,key
                }
            }'
        done | aria2c --conf-path='./configs/aria2.conf' -d "$cache_dir"

    for format in 'domain' 'ipv4' 'ipv6'; do
        list_name="${color}_${format}.txt"

        jq --arg color "$color" --arg format "$format" 'to_entries[] | select(.value.color == $color and .value.format == $format)' sources.json |
            jq -r -s 'from_entries | keys[] as $k | "($k)#(.[$k] | .rule)"' |
            while IFS=$'#' read -r key rule; do
                fpath=$(find -P -O3 "$cache_dir" -type f -name "$key*")

                case $fpath in
                *.tar.gz)
                    # Both Shallalist and Ut-capitole adhere to this format
                    # If any archives are added that do not, this line needs to change
                    tar -xOzf "$fpath" --wildcards-match-slash --wildcards '*/domains'
                    ;;
                *.zip) zcat "$fpath" ;;
                *) cat "$fpath" ;;
                esac |
                    gawk --sandbox -O -- "$rule" | # apply the regex rule
                    gawk '!x[$0]++' |              # filter duplicates out
                    gawk -v format="$format" -v color="$color" '{
                        switch (format) {
                        case "domain":
                            print $0 >> color "_domain.txt"
                            break
                        case "ipv4":
                            print "0.0.0.0 " $0 >> color "_ipv4.txt"
                            break
                        case "ipv6":
                            print ":: " $0 >> color "_ipv6.txt"
                            break
                        default:
                            break
                        }
                    }'
            done

        if test -f "$list_name"; then
            if [[ "$format" == "domain" ]]; then
                sort_list "$list_name" 1 "$cache_dir"
            else
                sort_list "$list_name" 2 "$cache_dir"
            fi

            if [[ "$color" == "black" ]]; then
                if test -f "white_${format}.txt"; then
                    grep -Fxvf "white_${format}.txt" "black_${format}.txt" | sponge "black_${format}.txt"
                fi

                tar -czf "black_$format.tar.gz" "black_$format.txt"
            fi

            if [[ "$format" == "domain" ]]; then
                gawk '{ print "0.0.0.0 " $0 }' "${color}_domain.txt" >"${color}_ipv4.txt"
                gawk '{ print ":: " $0 }' "${color}_domain.txt" >"${color}_ipv6.txt"
            fi
        fi
    done
done


Get this bounty!!!

#StackBounty: #18.04 #bash #python #systemd Configuring systemd to manage a service on boot

Bounty: 50

I have a script that downloads an image from a web page and sets it as my desktop wallpaper. The code works fine but I am having trouble getting it to run on boot.

I am trying to do this by configuring systemd to manage a service. (crontab didn’t work for me when rebooting but that’s another issue).

I made a file /etc/systemd/system/apod.service:

[Unit]
Description=Set APOD as Desktop

[Install]
WantedBy=multi-user.target

[Unit]
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/bin/bash /home/me/apod.sh
Type=simple
User=me
Group=me
WorkingDirectory=/home/me
Restart=on-failure

But when I boot it doesn’t seem to work. If I check systemctl status apod, I see:

Jun 04 20:55:55 me-XPS-15-9500 systemd[1]: Started Set APOD as Desktop.
Jun 04 20:55:57 me-XPS-15-9500 gsettings[1598]: failed to commit changes to dconf: Could not connect: No such file or directory

Yet if I just manually run /bin/bash /home/me/apod.sh then it works perfectly.

Any suggestions welcome. I’m running 18.04.


For completeness:

The bash script:

#!/bin/sh                                                                                                 

export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(id -u)/bus"

python /home/me/apod.py

/usr/bin/gsettings set org.gnome.desktop.background picture-uri "file:///home/me/Downloads/apod.jpg"

The python script it calls:

from bs4 import BeautifulSoup as BSHTML
import requests
import subprocess
import urllib2
page = urllib2.urlopen('https://apod.nasa.gov/apod/astropix.html')
soup = BSHTML(page,features="html.parser")
images = soup.findAll('img')

url = 'https://apod.nasa.gov/apod/'+images[0]['src']
r = requests.get(url, allow_redirects=True)
with open('/home/me/Downloads/apod.jpg',"w") as f:
            f.write(r.content)


Get this bounty!!!

#StackBounty: #bash #shell #sudo Providing Sudo Password in bash Script

Bounty: 50

I am trying to write a bash script to login to ssh and login to sudo

SSH login works fine , but i am facing problem while logging to sudo.This is my command

ssh -tt empId@env.linuxbox.com << EOF
echo "Passwordn" | sudo -S su - airbust 
cd ../builds
EOF

i)I am getting the error as Empty Password is provided

ii)The userName which i provided is ‘airbust’ whereas its taking userName as ’empId’

S su - airbustpost-lapp1941(Linux-RENO::TEST):/tmp]$ echo "Passwordn" | sudo - 
[sudo] password for : Sorry, try again.
[sudo] password for empId: 
sudo: no password was provided
sudo: 1 incorrect password attempt


Get this bounty!!!

#StackBounty: #bash #shell #dialog Has dialog's –checklist box type a possibility for horizontal srolling or n (newline) interpre…

Bounty: 50

I use dialog version: 1.3-20210324

Given:

LIST_ITEMS=(
  "1." "A ... very ... very ... looooong ... Informationtext" "off"
  "2." "A ... very ... very ... looooong ... Informationtext" "off"
                            ...                                     )
dialog ... --checklist "checklistName" 0 0 0 "${LIST_ITEMS[@]}" 

Result:
A classical dialog checklist with the Informationtext not visible entirely.

Missing Goal:
I need a way to scroll horizontally or linewrap the Informationtext.

Edit 1
@Thomas Dickey mentioned to reformat the data, that is no option for me because either way i do not have enough horizontal space. So as mentioned in my missing Goal section the real question remains. Anyways, thanks for your thought Thomas!
Any help is very welcome, thanks in advance!


Get this bounty!!!

#StackBounty: #bash #shell #dialog Has dialog's –checklist box type a possibility for horizontal srolling or n (newline) interpre…

Bounty: 50

I use dialog version: 1.3-20210324

Given:

LIST_ITEMS=(
  "1." "A ... very ... very ... looooong ... Informationtext" "off"
  "2." "A ... very ... very ... looooong ... Informationtext" "off"
                            ...                                     )
dialog ... --checklist "checklistName" 0 0 0 "${LIST_ITEMS[@]}" 

Result:
A classical dialog checklist with the Informationtext not visible entirely.

Missing Goal:
I need a way to scroll horizontally or linewrap the Informationtext.

Edit 1
@Thomas Dickey mentioned to reformat the data, that is no option for me because either way i do not have enough horizontal space. So as mentioned in my missing Goal section the real question remains. Anyways, thanks for your thought Thomas!
Any help is very welcome, thanks in advance!


Get this bounty!!!

#StackBounty: #bash #shell #dialog Has dialog's –checklist box type a possibility for horizontal srolling or n (newline) interpre…

Bounty: 50

I use dialog version: 1.3-20210324

Given:

LIST_ITEMS=(
  "1." "A ... very ... very ... looooong ... Informationtext" "off"
  "2." "A ... very ... very ... looooong ... Informationtext" "off"
                            ...                                     )
dialog ... --checklist "checklistName" 0 0 0 "${LIST_ITEMS[@]}" 

Result:
A classical dialog checklist with the Informationtext not visible entirely.

Missing Goal:
I need a way to scroll horizontally or linewrap the Informationtext.

Edit 1
@Thomas Dickey mentioned to reformat the data, that is no option for me because either way i do not have enough horizontal space. So as mentioned in my missing Goal section the real question remains. Anyways, thanks for your thought Thomas!
Any help is very welcome, thanks in advance!


Get this bounty!!!

#StackBounty: #bash #shell #dialog Has dialog's –checklist box type a possibility for horizontal srolling or n (newline) interpre…

Bounty: 50

I use dialog version: 1.3-20210324

Given:

LIST_ITEMS=(
  "1." "A ... very ... very ... looooong ... Informationtext" "off"
  "2." "A ... very ... very ... looooong ... Informationtext" "off"
                            ...                                     )
dialog ... --checklist "checklistName" 0 0 0 "${LIST_ITEMS[@]}" 

Result:
A classical dialog checklist with the Informationtext not visible entirely.

Missing Goal:
I need a way to scroll horizontally or linewrap the Informationtext.

Edit 1
@Thomas Dickey mentioned to reformat the data, that is no option for me because either way i do not have enough horizontal space. So as mentioned in my missing Goal section the real question remains. Anyways, thanks for your thought Thomas!
Any help is very welcome, thanks in advance!


Get this bounty!!!

#StackBounty: #bash #windows #display How to control screens on several Windows computers placed around a room?

Bounty: 500

I want to setup read-only screens around a room that displays different messages at different times. A single laptop would run a BASH script that checks the times, and decides which messages to send to which display, and when. This could either be text only (maybe a console, but with a huge font) or pictures, whichever is possible will be fine. A very basic script might look something like this:

#!/bin/bash
echo "Good morning!" > /dev/screen1
echo "Please stack the blocks as high as you can!" > /dev/screen2

My workplace has many unused Lenovo Yogas, so I could fold back the keyboard and mount seven or eight of them around the room with just the screens showing. I don’t think the workplace would allow me to remove Windows 10 from these devices.

  • Text or pictures, either is fine.
  • The Yogas do have HDMI ports.
  • People in the room won’t interact with the screens, they display information only.

The question is, how do I get text (or optionally pictures) from a BASH script on a laptop running Linux to display on these different messages or pictures on the Windows screens scattered around the room?


Get this bounty!!!

#StackBounty: #bash #shell #dialog Has dialog's –checklist box type a possibility for horizontal srolling or n (newline) interpre…

Bounty: 50

I use dialog version: 1.3-20210324

Given:

LIST_ITEMS=(
  "1." "A ... very ... very ... looooong ... Informationtext" "off"
  "2." "A ... very ... very ... looooong ... Informationtext" "off"
                            ...                                     )
dialog ... --checklist "checklistName" 0 0 0 "${LIST_ITEMS[@]}" 

Result:
A classical dialog checklist with the Informationtext not visible entirely.

Missing Goal:
I need a way to scroll horizontally or linewrap the Informationtext.

Edit 1
@Thomas Dickey mentioned to reformat the data, that is no option for me because either way i do not have enough horizontal space. So as mentioned in my missing Goal section the real question remains. Anyways, thanks for your thought Thomas!
Any help is very welcome, thanks in advance!


Get this bounty!!!

#StackBounty: #bash #shell #dialog Has dialog's –checklist box type a possibility for horizontal srolling or n (newline) interpre…

Bounty: 50

I use dialog version: 1.3-20210324

Given:

LIST_ITEMS=(
  "1." "A ... very ... very ... looooong ... Informationtext" "off"
  "2." "A ... very ... very ... looooong ... Informationtext" "off"
                            ...                                     )
dialog ... --checklist "checklistName" 0 0 0 "${LIST_ITEMS[@]}" 

Result:
A classical dialog checklist with the Informationtext not visible entirely.

Missing Goal:
I need a way to scroll horizontally or linewrap the Informationtext.

Edit 1
@Thomas Dickey mentioned to reformat the data, that is no option for me because either way i do not have enough horizontal space. So as mentioned in my missing Goal section the real question remains. Anyways, thanks for your thought Thomas!
Any help is very welcome, thanks in advance!


Get this bounty!!!