#StackBounty: #python #python-3.x #console #linux #installer Systemd service configuration helper script

Bounty: 50

I’ve written a script that semi-automates the process of configuring/creating a new systemd service.

The script is working perfectly well, however, I’ve had some trouble with the styling and readability, as well as with the argparse logic (argument parsing works well, but there is just too much to handle manually and not automatically with argparse, because I have many arguments and they have different requirements) – I don’t know if what I’ve done is best practice, because to me right now, it looks like a very poor idea.

The script right now loads a default schema (template) and allows the user to interactively configure a service using the options that are present in that template.

Notes: Version of Python used is 3.5.
There are some single cases of use where bugs might appear, however, the script is working correctly overall and the existence of such possible bugs should be ignored for the sake of this question, as they are not for here.

Available arguments for the script:

-c/--schema: Just loads a different template configuration to use 
for the interactive "asking" process. Takes the path to that template.

-s/--short: Uses a default schema (template) that is shorter than the 
original one (has less and more basic options)

-x/--extended: Uses a default schema (template) that is longer than the 
original one (has more options that are more complex)

-d/--directory: The directory in which the script should 
save the service unit file, by default "/etc/systemd/system".

--delete: Deletes the service unit file of the specified service name and 
disables/stops that service. If used, should be the only argument used 
along with service_name.

--edit: Edits the service unit file (opens an editor) 
of the specified service name. If used, should be the only argument used 
along with service_name.

-b/--build: Builds a default service configuration unit file as an example. 
Should be the ONLY argument used, even the 
positional argument service_name should not be used.

--info: Outputs information about the script - purpose, maintainer, etc. 
Again should be the ONLY argument used.

service_name: The positional argument which tells what is the name 
of the service that has to be configured/edited/deleted. 
Doesn't have to be used with --build/--info.

The script:

#!/usr/bin/env python3

"""
This is a helper script that semi-automates the process
of configuring/editing systemd services.

Examples --
Create a service using a short preset configuration:
sudo ./service-config.py --short some_service_name

Create a service using a custom preset configuration template:
sudo ./service-config.py -c some_schema some_service_name

Edit an existing service:
sudo ./service-config.py --edit some_service_name

# To do list:
1. Document the script.
2. Fix permission issues with some of the arguments.
3. Add a configuration file for the script for 
   some of the default values and options. (?)
4. Allow the user to choose an editor.

NOTES:
1. This script requires root privileges for most use-cases.
2. Skipping a configuration option (just pressing enter) without
   supplying any value will just tell the script to skip that option.
"""

# IMPORTS:

import argparse
import configparser
import datetime
import os
import subprocess
import sys
import time

from collections import OrderedDict

# DEFINING CONSTANTS:

VERSION             = "0.4"
MAINTAINER_NICK     = "..."
MAINTAINER_EMAIL    = "...@gmail.com"
TRACE               = True
SCHEMA              = "schemas/service-config"
SCHEMA_SHORT        = "schemas/short_service-config"
SCHEMA_EXTENDED     = "schemas/extended_service-config"
CONFIG              = None # for future uses
OUTPUT_DIR          = "/etc/systemd/system"

# ERRORS:

USER_ABORT          = 5
ARGPARSE_ERR        = 6
CONFIGURATION_ERR   = 7
GLOBAL_ERR          = 8
SCHEMA_ERR          = 9
SYSTEMD_ERR         = 10
UID_ERROR           = 11
FINISH_ERROR        = 12

# DEFAULTS:

DEFAULT_EDITOR           = "vim"
DEFAULT_BUILD_SCHEMA     = "schemas/default-schema"

DEFAULT_DESCRIPTION      = "Example"
DEFAULT_AFTER            = "network.target"
DEFAULT_TYPE             = "simple"
DEFAULT_USER             = "root"
DEFAULT_GROUP            = "root"
DEFAULT_EXEC_START       = "/bin/true"
DEFAULT_EXEC_STOP        = "/bin/true"
DEFAULT_KILL_MODE        = "control-group"
DEFAULT_KILL_SIGNAL      = "SIGTERM"
DEFAULT_PID_FILE         = "/run/service.pid"
DEFAULT_RESTART          = "on-failure"
DEFAULT_RESTART_SEC      = "2"
DEFAULT_TIMEOUT_STOP_SEC = "5"
DEFAULT_DYNAMIC_USER     = "no"
DEFAULT_ENVIRONMENT_FILE = "/etc/service/env"
DEFAULT_STANDARD_OUTPUT  = "journal"
DEFAULT_STANDARD_ERROR   = "journal"
DEFAULT_WANTED_BY        = "multi-user.target"

# COLORS AND Formatting:

def tty_supports_ansi():
    """Checks whether the terminal used supports ANSI codes."""

    for handle in [sys.stdout, sys.stderr]:
        if ((hasattr(handle, "isatty") and handle.isatty()) or
            ('TERM' in os.environ and os.environ['TERM'] == "ANSI")):
            return True
        else:
            return False

class Formatting:
    """
    A class containing constants with most Formatting/basic colors
    for Unix-based terminals and vtys.
    Does NOT work on Windows cmd, PowerShell, and their varieties!
    """

    RESET            = "33[0m"
    BOLD             = "33[1m"
    DIM              = "33[2m"
    ITALIC           = "33[3m"
    UNDERLINE        = "33[4m"
    BLINK            = "33[5m" # Doesn't work on some terminals.
    INVERT           = "33[7m"
    HIDDEN           = "33[8m"
    FG_DEFAULT       = "33[39m"
    FG_BLACK         = "33[30m"
    FG_RED           = "33[31m"
    FG_GREEN         = "33[32m"
    FG_YELLOW        = "33[33m"
    FG_BLUE          = "33[34m"
    FG_MAGENTA       = "33[35m"
    FG_CYAN          = "33[36m"
    FG_LIGHT_GRAY    = "33[37m"
    FG_DARK_GRAY     = "33[90m"
    FG_LIGHT_RED     = "33[91m"
    FG_LIGHT_GREEN   = "33[92m"
    FG_LIGHT_YELLOW  = "33[93m"
    FG_LIGHT_BLUE    = "33[94m"
    FG_LIGHT_MAGENTA = "33[95m"
    FG_LIGHT_CYAN    = "33[96m"
    FG_WHITE         = "33[97m"
    BG_DEFAULT       = "33[49m"
    BG_BLACK         = "33[40m"
    BG_RED           = "33[41m"
    BG_GREEN         = "33[42m"
    BG_YELLOW        = "33[43m"
    BG_BLUE          = "33[44m"
    BG_MAGENTA       = "33[45m"
    BG_CYAN          = "33[46m"
    BG_LIGHT_GRAY    = "33[47m"
    BG_DARK_GRAY     = "33[100m"
    BG_LIGHT_RED     = "33[101m"
    BG_LIGHT_GREEN   = "33[102m"
    BG_LIGHT_YELLOW  = "33[103m"
    BG_LIGHT_BLUE    = "33[104m"
    BG_LIGHT_MAGENTA = "33[105m"
    BG_LIGHT_CYAN    = "33[106m"
    BG_WHITE         = "33[107m"

    def __init__(self):
        self.is_supported = tty_supports_ansi()

    def ansi(self, ansi_key):
        """
        The format method for this class. Returns the proper
        chosen formatting if it is supported, else does nothing.
        Takes ansi_key as argument, where ansi_key is one of the
        defined constants in the class.
        """
        if self.is_supported:
            return getattr(self, ansi_key)
        else:
            return ""

# The formatting/color class handler variable

FTY = Formatting()

# Code starts from here:

def printf(text, f="RESET", **kwargs):
    """
    A print function with formatting.
    Always prints on stdout.
    As arguments takes:
    1. string (text to print)
    2. formatting type (bold, italic, etc.)
    3+. kwargs passed to the print function.
    """

    f = FTY.ansi(f.upper())

    print("{}{}".format(f, text), file=sys.stdout, **kwargs)


def print_info():
    """Print information about the script."""

    printf("This is a helper script for configuring systemd services.", f="bold")
    printf("{}Maintainer: {}{}".format(FTY.ansi("FG_GREEN"), FTY.ansi("RESET"), MAINTAINER_NICK))
    printf("{}Email: {}{}".format(FTY.ansi("FG_GREEN"), FTY.ansi("RESET"), MAINTAINER_EMAIL))

    sys.exit(0)

def parse_arg():
    """Get user arguments and configure them."""

    parser = argparse.ArgumentParser(description="Systemd services configuration script")
    no_pos = parser.add_mutually_exclusive_group()
    schema = parser.add_mutually_exclusive_group()
    exclus = parser.add_mutually_exclusive_group()
    exclus.add_argument("-c",
                        "--schema",
                        help="Choose a custom schema and load defaults from it.",
                        type=str,
                        default=SCHEMA)
    exclus.add_argument("--edit",
                        help="Directly edit a systemd unit file.",
                        action="store_true",
                        default=False)
    no_pos.add_argument("--info",
                        help="Show information about the script.",
                        action="store_true",
                        default=False)
    no_pos.add_argument("-b",
                        "--build",
                        help="Builds a default schema in schemas/default-schema",
                        action="store_true",
                        default=False)
    schema.add_argument("-s",
                        "--short",
                        help="Use a short configuration schema.",
                        action="store_true",
                        default=False)
    schema.add_argument("-x",
                        "--extended",
                        help="Use a long configuration schema.",
                        action="store_true",
                        default=False)
    exclus.add_argument("-d",
                        "--directory",
                        help="Output directory for the service unit file.",
                        type=str,
                        default=OUTPUT_DIR)
    exclus.add_argument("--delete",
                        help="Delete the specified service's configuration file.",
                        action="store_true",
                        default=False)
    no_pos.add_argument("service_name",
                        help="The name of the service to configure/edit.",
                        type=str,
                        nargs='?')

    try:
        args = parser.parse_args()
        check_parser_opts(args)
    except argparse.ArgumentError:
        print("Error: An error occured while parsing your arguments.", file=sys.stderr)
        sys.exit(ARGPARSE_ERR)

    return args

def check_parser_opts(args):
    """
    Check if all supplied arguments are used correctly.
    Returns an error if the combination of 
    supplied arguments is illegal.
    For arguments that are supposed to be 
    single checks if any other arguments
    are used (besides directory and schema, 
    since they have default values already).
    """

    all_args = [
        'build',
        'service_name',
        'info',
        'short',
        'extended',
        'delete',
        'directory',
        'edit',
        'schema'
    ]

    error = False

    # --build is supposed to be used alone.
    if args.build:
        for arg in all_args:
            if (arg is not 'build'     and 
                arg is not 'directory' and 
                arg is not 'schema'):
                value = getattr(args, arg)
            else:
                value = None
            if value:
                print("The argument -b/--build cannot be used with {}.".format(arg))
                error = True

    # --info is supposed to be used alone.
    if args.info:
        for arg in all_args:
            if (arg is not 'info'      and 
                arg is not 'directory' and 
                arg is not 'schema'):
                value = getattr(args, arg)
            else:
                value = None
            if value:
                print("The argument --info cannot be used with {}.".format(arg))
                error = True

    # --delete is supposed to be used only with service_name.
    if args.delete:
        for arg in all_args:
            if (arg is not 'delete'    and 
                arg is not 'directory' and 
                arg is not 'schema'    and 
                arg is not 'service_name'):
                value = getattr(args, arg)
            else:
                value = None
            if value:
                print("The argument --delete cannot be used with {}.".format(arg))
                error = True

    # --edit is supposed to be used only with service_name.
    if args.edit:
        for arg in all_args:
            if (arg is not 'edit'      and 
                arg is not 'directory' and 
                arg is not 'schema'    and 
                arg is not 'service_name'):
                value = getattr(args, arg)
            else:
                value = None
            if value:
                print("The argument --edit cannot be used with {}.".format(arg))
                error = True

    if error:
        printf("{}Error: wrong argument usage, aborting.".format(FTY.ansi("FG_RED")), f="bold")
        sys.exit(ARGPARSE_ERR)

def get_fragment_path(service):
    """
    Returns the path of the systemd service's unit configuration file.
    """

    # Extremely ugly (imo) multiline statement
    sysctl_out = subprocess.check_output("systemctl show {} -p FragmentPath".format(service), 
        shell=True)
    filename = sysctl_out.decode('utf-8').strip().split('=')[1]

    return filename

def edit(service, manual=False, finish=True):
    """
    Open the service's systemd service 
    unit configuration file for editing.
    """

    # Check if destination is already set to the full path.
    if manual:
        file = service
    else:
        file = get_fragment_path(service)

    # Open vim to edit the configuration file. This is a TODO.
    with subprocess.Popen(["{} {}".format(DEFAULT_EDITOR, file)], shell=True) as command:
        subprocess.Popen.wait(command)

    if finish: 
        finish(file, mode="edit")

def delete(service):
    """
    Deletes the given service configuration file, 
    stops and disables the service.
    """

    # Get the destination of the service unit file.
    destination = get_fragment_path(service)

    # Ask the user for confirmation if it's a system service.
    if destination.startswith("/lib/systemd/system"):
        print("This is not a user-configured service, do you want to delete it anyway? [y/N]: ")
        force_delete = input()
        if not (force_delete and (force_delete.lower() == 'y' or force_delete.lower() == 'yes')):
            print("Aborting...")
            sys.exit(0)

    # Stop, disable, and delete the service.
    print("Deleting service...")
    sysctl_service(service, "stop")
    sysctl_service(service, "disable")
    os.remove(destination)
    service_reload()
    print("Deleted service.")

def setup(args):
    """
    Check systemd version available on the host to confirm compability.
    Also checks whether we have permissions to use
    most of the script's functionality.
    """

    # Get the systemd version to confirm compability. This is for a future update.
    try:
        systemd_version = subprocess.check_output('systemd --version', shell=True)
        systemd_version = int(systemd_version.strip().split()[1])
    except subprocess.CalledProcessError:
        print("Systemd isn't working on your system. Why even use this script?", file=sys.stderr)
        sys.exit(SYSTEMD_ERR)

    if os.getuid() > 0 and not args.build and args.directory == OUTPUT_DIR:
        # Extremely ugly (imo) multiline statement
        printf("{}Insufficient permissions. "
               "You have to run the script as root (with sudo).".format(
                FTY.ansi("FG_LIGHT_RED")), f="bold", file=sys.stderr)
        sys.exit(UID_ERROR)

    return systemd_version

def build():
    """
    Build a default example unit configuration schema, 
    using default constants.
    """

    if os.path.exists(DEFAULT_BUILD_SCHEMA):
        print("Error: {} already exists.".format(DEFAULT_BUILD_SCHEMA), file=sys.stderr)
        sys.exit(SCHEMA_ERR)
    else:
        schema = configparser.ConfigParser()
        schema.optionxform = str
        schema['Unit'] = OrderedDict(
            Description     = DEFAULT_DESCRIPTION,
            After           = DEFAULT_AFTER
        )

        schema['Service'] = OrderedDict(
            Type            = DEFAULT_TYPE,
            ExecStart       = DEFAULT_EXEC_START,
            ExecStop        = DEFAULT_EXEC_STOP,
            Restart         = DEFAULT_RESTART,
            RestartSec      = DEFAULT_RESTART_SEC,
            User            = DEFAULT_USER,
            Group           = DEFAULT_GROUP,
            PIDFile         = DEFAULT_PID_FILE,
            EnvironmentFile = DEFAULT_ENVIRONMENT_FILE,
            KillMode        = DEFAULT_KILL_MODE,
            KillSignal      = DEFAULT_KILL_SIGNAL,
            TimeoutStopSec  = DEFAULT_TIMEOUT_STOP_SEC,
            StandardOutput  = DEFAULT_STANDARD_OUTPUT,
            StandardError   = DEFAULT_STANDARD_ERROR,
            DynamicUser     = DEFAULT_DYNAMIC_USER
        )

        schema['Install'] = OrderedDict(
            WantedBy        = DEFAULT_WANTED_BY
        )

        with open(DEFAULT_BUILD_SCHEMA, 'w+') as schemafile:
            schema.write(schemafile)

        finish(DEFAULT_BUILD_SCHEMA, mode="build")

def load_schema(schema):
    """
    Read the service unit configuration file and load it.
    """

    config_dict = {}

    config = configparser.ConfigParser()
    config.optionxform = str
    config.read(schema)

    return config

def parse_config(cfg):
    """
    Parse the configuration file and return it as dictionaries.
    """

    config         = argparse.Namespace(**OrderedDict(cfg))
    config.Unit    = OrderedDict(config.Unit)
    config.Service = OrderedDict(config.Service)
    config.Install = OrderedDict(config.Install)

    return config

def write_config(cfg, destination):
    """
    Save the unit configuration file to the destination.
    """

    config = configparser.ConfigParser()
    config.optionxform = str
    config['Unit'] = cfg.Unit
    config['Service'] = cfg.Service
    if cfg.Install:
        config['Install'] = cfg.Install
    with open(destination, 'w') as unitfile:
        config.write(unitfile)
        unitfile.write("# Automatically generated by service-config.n")

def user_configuration(config):
    """
    Let the user interactively configure the unit file.
    """

    user_config = config

    # Ask for the [Unit] section's keys.
    printf("{}[Unit] section configuration:".format(FTY.ansi("FG_YELLOW")), f="bold")
    for key in config.Unit:
        printf("{}{}={}".format(FTY.ansi("FG_GREEN"), key, FTY.ansi("RESET")), f="bold", end="")
        value = input()
        user_config.Unit[key] = value

    # Ask for the [Service] section's keys.
    print()
    printf("{}[Service] section configuration:".format(FTY.ansi("FG_BLUE")), f="bold")
    for key in config.Service:
        printf("{}{}={}".format(FTY.ansi("FG_GREEN"), key, FTY.ansi("RESET")), f="bold", end="")
        value = input()
        user_config.Service[key] = value

    # Ask for the [Install] section's keys.
    print()
    printf("{}[Install] section configuration:".format(FTY.ansi("FG_MAGENTA")), f="bold")
    for key in config.Install:
        printf("{}{}={}".format(FTY.ansi("FG_GREEN"), key, FTY.ansi("RESET")), f="bold", end="")
        value = input()
        user_config.Install[key] = value

    # Clear the dictionaries from any empty keys.
    user_config.Unit = {k: v for k, v in user_config.Unit.items() if v}
    user_config.Service = {k: v for k, v in user_config.Service.items() if v}
    user_config.Install = {k: v for k, v in user_config.Install.items() if v}

    return user_config

def service_reload():
    """
    A simple daemon-reload wrapper.
    """

    subprocess.call('systemctl daemon-reload', shell=True)

def sysctl_service(service, action):
    """
    A simple systemctl wrapper for service management.
    """

    subprocess.call('systemctl {} {}'.format(action, service), shell=True)

def finish(destination, mode="create"):
    """
    Checks whether the file has been saved successfully and exits.
    """

    if os.path.exists(destination):
        if mode == "create":
            print("{}Service created successfully.".format(FTY.ansi("FG_GREEN")))
        elif mode == "edit":
            print("{}Service edited successfully.".format(FTY.ansi("FG_YELLOW")))
        elif mode == "build":
            print("{}Default schema built successfully.".format(FTY.ansi("FG_BLUE")))
        sys.exit(0)
    else:
        print("The script failed to finish successfully.")
        sys.exit(FINISH_ERROR)

def main():
    """
    The main function that handles the program.
    """

    # Get the parsed arguments.
    args = parse_arg()

    # Check the version of systemd and check permissions.
    systemd_version = setup(args)

    # Exit if service_name contains illegal characters.
    if args.service_name:
        if 'x00' in args.service_name or '/' in args.service_name:
            print("Service name contains symbols that are not allowed.")
            sys.exit(ARGPARSE_ERR)

    if args.delete:
        delete(args.service_name)
        sys.exit(0)
    if args.info:
        print_info()
    if args.build:
        build()
    if args.edit:
        edit(args.service_name)
    if args.short:
        args.schema = SCHEMA_SHORT
        print("Using short schema configuration.")
    if args.extended:
        args.schema = SCHEMA_EXTENDED
        print("Using extended schema configuration.")

    # Load and parse the unit configuration schema.
    schema = load_schema(args.schema)
    config = parse_config(schema)

    # Start interactive configuration, aborts on CTRL-C/CTRL-D.
    try:
        user_config = user_configuration(config)
    except (EOFError, KeyboardInterrupt):
        print("nAborting.")
        sys.exit(USER_ABORT)

    # Check whether the supplied service name ends with .service.
    if not args.service_name.endswith('.service'):
        args.service_name = args.service_name + '.service'

    # Save the configured unit file to the destination directory.
    destination = os.path.join(args.directory, args.service_name)
    write_config(user_config, destination)


    # Interactive section:

    print("Do you want to manually edit the new configuration? [y/N]: ", end="")
    manual = input()

    if manual and manual.lower() == "y":
        print("Opening editor...")
        edit(destination, manual=True, finish=False)
    else:
        print("The configuration file won't be edited.")

    # Allow these options only if we have permissions for them.
    if os.getuid() == 0:
        print("Do you want to enable the service? [y/N]: ", end="")
        enable = input()

        if enable and enable.lower() == "y":
            print("Enabling service...")
            service_reload()
            sysctl_service(args.service_name, "enable")
            print("Service enabled.")
        else:
            print("Service won't be enabled.")


        print("Do you want to start the service? [Y/n]: ", end="")
        start = input()

        if not start or (start and start.lower() == "y"):
            print("Starting service...")
            service_reload()
            sysctl_service(args.service_name, "start")
            print("Service started.")
        else:
            print("Service won't be started.")

    elif os.getuid() > 0:
        # Extremely ugly (imo) multiline statement
        print("{}No permissions to enable/start service. "
              "Need to run with root privileges.".format(FTY.ansi("FG_RED")))

    finish(destination)


# Name guard for main process.
if __name__ == "__main__":
    # Don't use global error handling if TRACE is set to True.
    if TRACE:
        main()
    elif not TRACE:
        try:
            main()
        except Exception as error:
            print("A global exception has been caught.", file=sys.stderr)
            print(err, file=sys.stderr)
            sys.exit(GLOBAL_ERR)

I want this script to be reviewed for styling/formatting and readability. I do want it to be fairly easy to read it, since I won’t be keeping it only to myself.

Questions:

  1. I’ve had issues with styling. There are some “extremely ugly multiline statements” that I’ve made in order to comply with the 100 characters limit I’ve chosen. Such lines are commented, I do believe them to be ugly. Is the way I’ve made them really best practice or is there a better way?
  2. The Formatting class. I have a class that I use to format/colorize the output of my script, however, the ANSI color codes do not work on every terminal, thus I’ve made the class the way it is. Is it good practice to have such a class with constants and return the values of those constants by using a method (that allows me to check if the terminal supports ANSI)? Is the way I’ve gone about coloring/formatting the output a good idea at all, having so many constants for that in my main program and also having a variable to initialize such a class?
  3. The way I check for the argument usage. Sure, I have mutually exclusive groups, but those aren’t nearly enough to help me achieve what I need. So I’ve made a function check_parser_opts(args) that checks whether the arguments are used correctly with a bunch of if statements. Is the way I’ve done it proper, how could I go about this? Or should I just try to simplify my arguments and not have this many?
  4. I’ve always wondered how should my functions be ordered. Which function should be in which part of the file, after which function, and etc. Is there any “proper” way to order your functions? Currently I just have my main() function at the bottom and I just put everything else above it, without really thinking too much.

Note: For enthusiasts that are interested in the progress of this script, you can follow the github repo.


Get this bounty!!!

#StackBounty: #python #python-3.x #lexer Lexer for Apache logs in Python

Bounty: 100

I recently learned Python using the book by Mark Summerfeld and I am now doing a few katas of mine when I learn a new language. This helps to get the “how the language works” and I wrote a simple lexer for Apache logs.

An example of log records are

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846
64.242.88.10 - - [07/Mar/2004:16:06:51 -0800] "GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1" 200 4523
64.242.88.10 - - [07/Mar/2004:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291

and program I wrote reads all values as tokens and output them using | as a separator, as in

64.242.88.10|-|-|07/Mar/2004:16:05:49 -0800|GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1|401|12846
64.242.88.10|-|-|07/Mar/2004:16:06:51 -0800|GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1|200|4523
64.242.88.10|-|-|07/Mar/2004:16:10:02 -0800|GET /mailman/listinfo/hsdivision HTTP/1.1|200|6291

The lexer can be used for more general usage though.

Here is my code and I would love to read your comments and suggestions about how to improve it. I have also raised a few specific questions on certain topics.

import collections
import io
import sys

LEXER_SKIP_WHITESPACE=0
LEXER_READ_QUOTED_STRING=1
LEXER_READ_BRACKETED_STRING=2
LEXER_READ_WORD=3

class Location:
    def __init__(self, name=None, pos=0, line=1, col=0):
        self.name = name or "<input>"
        self.line = line
        self.col = col
        self.pos = pos
    def update(self, c):
        self.pos += 1
        if c == "n":
            self.line += 1
            self.col = 0
        else:
            self.col += 1
    def __repr__(self):
        return str.format("Location({}, {}, {}, {})", repr(self.name), repr(self.pos), repr(self.line), repr(self.col))
    def __str__(self):
        return str.format("{}: {}: line {}, column {}", self.name, self.pos, self.line, self.col)

def readchar(inputchannel, location):
    while True:
        maybechar = inputchannel.read(1)
        if maybechar == '':
            return None
        else:
            location.update(maybechar)
            yield maybechar

def readtoken(inputchannel, location):
    state = LEXER_SKIP_WHITESPACE
    token = ''
    for nextchar in readchar(inputchannel, location):
        if state is LEXER_SKIP_WHITESPACE:
            if nextchar == "n":
                yield "n"
                continue
            elif nextchar.isspace():
                continue
            elif nextchar == '"':
                state = LEXER_READ_QUOTED_STRING
                continue
            elif nextchar == '[':
                state = LEXER_READ_BRACKETED_STRING
                continue
            else:
                state = LEXER_READ_WORD
                token += nextchar
                continue
        elif state is LEXER_READ_QUOTED_STRING:
            if nextchar == '"':
                yield token
                token = ''
                state = LEXER_SKIP_WHITESPACE
                continue
            else:
                token += nextchar
                continue
        elif state is LEXER_READ_BRACKETED_STRING:
            if nextchar == ']':
                yield token
                token = ''
                state = LEXER_SKIP_WHITESPACE
                continue
            else:
                token += nextchar
                continue
        elif state is LEXER_READ_WORD:
            if nextchar == "n":
                yield token
                token = ''
                state = LEXER_SKIP_WHITESPACE
                yield "n"
                continue
            elif nextchar.isspace():
                yield token
                token = ''
                state = LEXER_SKIP_WHITESPACE
                continue
            else:
                token += nextchar
                continue
        else:
            raise Error("Impossible lexer state.")
    if state is LEXER_SKIP_WHITESPACE:
        return None
    elif state is LEXER_READ_QUOTED_STRING:
        raise Error("End of character stream in quoted string.")
    elif state is LEXER_READ_BRACKETED_STRING:
        raise Error("End of character stream in quoted string.")
    elif state is LEXER_READ_WORD:
        yield token
        return None
    else:
        raise Error("Impossible lexer state.")


class Lexer:
    def __init__(self, inputchannel, _location=None):
        self.location = _location or Location("<input>", 0, 1, 0)
        self.inputchannel = inputchannel
        self.buf = ''
        self.state = LEXER_SKIP_WHITESPACE

    def __iter__(self):
        return readtoken(self.inputchannel, self.location)

if __name__ == "__main__":
    sep = ''
    for token in Lexer(sys.stdin):
        if token == 'n':
            sys.stdout.write(token)
            sep = ''
        else:
            sys.stdout.write(sep)
            sys.stdout.write(token)
            sep = '|'

Question 1 – How to write lexers in Python?

I am well aware that there are tools similar to lex and yacc for Python that could have been used here, but that would defeat the purpose of learning how to write such a program in Python. I found it surprisingly hard.

My first misfortune is that Python does not do tail-elimination, so it is essentially forbidden to write a lexer as a set of mutually recursive functions. Mutually recursive functions are one of my favourite tools to do so, because it clearly singles out a specific state of the lexer (the recursive function it is in) and the transitions from this state to the other states, and makes it easy to test-case individually each of the transitions.

Since maybe not everybody is familiar with lexers based on mutually recursive functions here is the equivalent of the readtoken generator in OCaml. The beginning of read_token reads like “If you read a double quote, discard it and do read_quotedstring” and that function itself is defined later and does what on expects: it aggregates characters in a buffer until the next double quote and bless the result as a token.

let rec read_token f ((ax,buffer) as state) s =
  match Stream.peek s with
  | Some('"') -> Stream.junk s; read_quotedstring f state s
  | Some('[') -> Stream.junk s; read_timestamp f state s
  | Some(' ')
  | Some('t')-> Stream.junk s; read_token f state s
  | Some(c) -> read_simpleword f state s
  | None -> ax
and read_simpleword f ((ax,buffer) as state) s =
  match Stream.peek s with
  | Some('"')
  | Some('[')
  | Some(' ')
  | Some('t') ->
    let current_token = Buffer.contents buffer in
    Buffer.clear buffer;
    read_token f (f ax current_token, buffer) s
  | Some(c) ->
    Buffer.add_char buffer c;
    Stream.junk s;
    read_simpleword f state s
  | None ->
    ax
and read_quotedstring f ((ax,buffer) as state) s =
  match Stream.peek(s) with
  | Some('"') ->
    Stream.junk s;
    let current_token = Buffer.contents buffer in
    Buffer.clear buffer;
    read_token f (f ax current_token, buffer) s
  | Some(c) ->
    Stream.junk s;
    Buffer.add_char buffer c;
    read_quotedstring f state s
  | None ->
    failwith "End of stream within a quoted string"
and read_timestamp f ((ax,buffer) as state) s =
  match Stream.peek(s) with
  | Some(']') ->
    Stream.junk s;
    let current_token = Buffer.contents buffer in
    Buffer.clear buffer;
    read_token f (f ax (timestamp_to_iso current_token), buffer) s
  | Some(c) ->
    Stream.junk s;
    Buffer.add_char buffer c;
    read_timestamp f state s
  | None ->
    failwith "End of stream within a bracketed string"

My Python version defines a few states as symbolic constants as I would have done in C and then build a huge while loop keeping track of the transitions. I am not please with that implementation, because it does not give me any tool to manage the complexity of the lexer. Working with functions is very unpleasant because of the details of the scope of variables in Python. So what would be the idiomatic way to break down the lexer in small testable pieces, which would be important if I would decide to write a more complicate parser? Could the idea to represent lexer-states by objects be interesting?

Question 2. Is it right to treat stdin as a character stream?

Obviously I somehow did that but Python has no real character type and makes them look like length 1 strings. It feels to me that the approach of reading my input in chunks into “circular expandable buffers” and making copies of substrings of these chunks to generate my tokens would fit much more nicely in the language. Am I right?

Question 3. What is the all-purpose exception in Python?

This seems like a rather basic question but I am really failing to find a suitable answer in the documentation The Exception seems a poor choice, since it very general and make error identification and error handling rather complicated.

Question 4. Do generators return none?

Is it good style to return None in generators? Would pass also do?


Get this bounty!!!

#StackBounty: #python #python-3.x #mvc #asynchronous #url-routing Python (Sanic) Routing

Bounty: 50

I am new to Python and wanted to explore how I can associate routes with “controllers” with Sanic as a base. It works fine but “feels” a bit clunky but I can’t put my finger on why/how.

routes.py – Returning controller methods from a given route

from .index import IndexRoutes
from .clients import ClientRoutes
from Controllers.IndexController import *
from Controllers.ClientController import *

CLASSES = [
    IndexRoutes,
    ClientRoutes
]

def add_routes(app):
    routes = get_routes()

    for route in routes:
        for m in route['methods']:
            controller = m['controller']
            method = m['method']

            app.add_route(
                eval(controller + '.' + method),
                route['prefix'] + m['path'],
                methods=m['request_methods']
            )


def get_routes():
    routes = []

    for c in CLASSES:
        methods = getattr(c, 'ROUTE_METHODS')

        route = {
            'name': getattr(c, 'ROUTE_NAME'),
            'methods': [],
            'prefix': getattr(c, 'ROUTE_PREFIX')
        }

        for method in methods:
            route['methods'].append(getattr(c, method)())

        routes.append(route)

    return routes

client.py – Example of route definitions

class ClientRoutes:
    ROUTE_NAME = 'clients'
    ROUTE_METHODS = ['index', 'create', 'update', 'single', 'destroy']
    ROUTE_PREFIX = 'clients'


    def index():
        return {
            'controller': 'ClientController',
            'method': 'index',
            'path': '/',
            'request_methods': ['GET']
        }


    def create():
        return {
            'controller': 'ClientController',
            'method': 'create',
            'path': '/',
            'request_methods': ['POST']
        }


    def single():
        return {
            'controller': 'ClientController',
            'method': 'single',
            'path': '/<id>',
            'request_methods': ['GET']
        }


    def update():
        return {
            'controller': 'ClientController',
            'method': 'update',
            'path': '/<id>',
            'request_methods': ['PUT']
        }


    def destroy():
        return {
            'controller': 'ClientController',
            'method': 'destroy',
            'path': '/<id>',
            'request_methods': ['DELETE']
        }

ClientController.py – Example of controller

from sanic import response
from Database.Model import Model
from Database.Client import Client
from Helpers.transformers import to_dict

class ClientController:

    async def index(request):
        clients = Model(Client)
        return response.json(to_dict(clients.get()))

    async def create(request):
        data = request.json
        try:
            if 'name' not in data:
                raise KeyError('Name is required')
            else:
                if len(data['name']) < 1:
                    raise ValueError('Name is required')
        except KeyError as e:
            return response.json(str(e), 400)
        except ValueError as e:
            return response.json(str(e), 400)

        client = Model(Client)
        return response.json(to_dict(client.save(data)))

        # ... Other methods below, unnecessary for this example


Get this bounty!!!

#StackBounty: #python #python-3.x #mvc #url-routing Python (Sanic) Routing

Bounty: 50

I am new to Python and wanted to explore how I can associate routes with “controllers” with Sanic as a base. It works fine but “feels” a bit clunky but I can’t put my finger on why/how.

routes.py – Returning controller methods from a given route

from .index import IndexRoutes
from .clients import ClientRoutes
from Controllers.IndexController import *
from Controllers.ClientController import *

CLASSES = [
    IndexRoutes,
    ClientRoutes
]

def add_routes(app):
    routes = get_routes()

    for route in routes:
        for m in route['methods']:
            controller = m['controller']
            method = m['method']

            app.add_route(
                eval(controller + '.' + method),
                route['prefix'] + m['path'],
                methods=m['request_methods']
            )


def get_routes():
    routes = []

    for c in CLASSES:
        methods = getattr(c, 'ROUTE_METHODS')

        route = {
            'name': getattr(c, 'ROUTE_NAME'),
            'methods': [],
            'prefix': getattr(c, 'ROUTE_PREFIX')
        }

        for method in methods:
            route['methods'].append(getattr(c, method)())

        routes.append(route)

    return routes

client.py – Example of route definitions

class ClientRoutes:
    ROUTE_NAME = 'clients'
    ROUTE_METHODS = ['index', 'create', 'update', 'single', 'destroy']
    ROUTE_PREFIX = 'clients'


    def index():
        return {
            'controller': 'ClientController',
            'method': 'index',
            'path': '/',
            'request_methods': ['GET']
        }


    def create():
        return {
            'controller': 'ClientController',
            'method': 'create',
            'path': '/',
            'request_methods': ['POST']
        }


    def single():
        return {
            'controller': 'ClientController',
            'method': 'single',
            'path': '/<id>',
            'request_methods': ['GET']
        }


    def update():
        return {
            'controller': 'ClientController',
            'method': 'update',
            'path': '/<id>',
            'request_methods': ['PUT']
        }


    def destroy():
        return {
            'controller': 'ClientController',
            'method': 'destroy',
            'path': '/<id>',
            'request_methods': ['DELETE']
        }

ClientController.py – Example of controller

from sanic import response
from Database.Model import Model
from Database.Client import Client
from Helpers.transformers import to_dict

class ClientController:

    async def index(request):
        clients = Model(Client)
        return response.json(to_dict(clients.get()))

    async def create(request):
        data = request.json
        try:
            if 'name' not in data:
                raise KeyError('Name is required')
            else:
                if len(data['name']) < 1:
                    raise ValueError('Name is required')
        except KeyError as e:
            return response.json(str(e), 400)
        except ValueError as e:
            return response.json(str(e), 400)

        client = Model(Client)
        return response.json(to_dict(client.save(data)))

        # ... Other methods below, unnecessary for this example


Get this bounty!!!

#StackBounty: #python-3.x #matplotlib #pyqt5 Why Matplotlib style do not update facecolor PyQt5?

Bounty: 50

I am using matplotlib styles and I try to change the styles dynamically when the style is chosen in the combobox. I am using Matplotlib 2.2.3, Python 3.6.6, PyQt5, Windows 10. But when I choose the dark_background style, the figure facecolor and the axes facecolor do not change. Here an animation:
dark_bakground error
This is the code:

File IHMDrawDates.py generated with pyuic5:

from PyQt5 import QtCore, QtGui, QtWidgets

class Ui_MplMainWindow(object):
    def setupUi(self, MplMainWindow):
        MplMainWindow.setObjectName("MplMainWindow")
        MplMainWindow.resize(628, 416)
        self.centralwidget = QtWidgets.QWidget(MplMainWindow)
        self.centralwidget.setObjectName("centralwidget")
        self.gridLayout_2 = QtWidgets.QGridLayout(self.centralwidget)
        self.gridLayout_2.setObjectName("gridLayout_2")
        self.mpl = MplWidgetTest(self.centralwidget)
        sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Expanding, QtWidgets.QSizePolicy.Expanding)
        sizePolicy.setHorizontalStretch(0)
        sizePolicy.setVerticalStretch(0)
        sizePolicy.setHeightForWidth(self.mpl.sizePolicy().hasHeightForWidth())
        self.mpl.setSizePolicy(sizePolicy)
        self.mpl.setObjectName("mpl")
        self.gridLayout_2.addWidget(self.mpl, 0, 0, 1, 1)
        self.groupBox = QtWidgets.QGroupBox(self.centralwidget)
        self.groupBox.setMaximumSize(QtCore.QSize(95, 16777215))
        self.groupBox.setObjectName("groupBox")
        self.gridLayout = QtWidgets.QGridLayout(self.groupBox)
        self.gridLayout.setObjectName("gridLayout")
        self.buttonDrawDate = QtWidgets.QPushButton(self.groupBox)
        self.buttonDrawDate.setMaximumSize(QtCore.QSize(75, 16777215))
        self.buttonDrawDate.setObjectName("buttonDrawDate")
        self.gridLayout.addWidget(self.buttonDrawDate, 1, 0, 1, 1)
        self.buttonErase = QtWidgets.QPushButton(self.groupBox)
        self.buttonErase.setMaximumSize(QtCore.QSize(75, 16777215))
        self.buttonErase.setObjectName("buttonErase")
        self.gridLayout.addWidget(self.buttonErase, 2, 0, 1, 1)
        spacerItem = QtWidgets.QSpacerItem(20, 40, QtWidgets.QSizePolicy.Minimum, QtWidgets.QSizePolicy.Expanding)
        self.gridLayout.addItem(spacerItem, 3, 0, 1, 1)
        self.comboTema = QtWidgets.QComboBox(self.groupBox)
        self.comboTema.setObjectName("comboTema")
        self.gridLayout.addWidget(self.comboTema, 0, 0, 1, 1)
        self.gridLayout_2.addWidget(self.groupBox, 0, 1, 1, 1)
        MplMainWindow.setCentralWidget(self.centralwidget)
        self.menubar = QtWidgets.QMenuBar(MplMainWindow)
        self.menubar.setGeometry(QtCore.QRect(0, 0, 628, 21))
        self.menubar.setObjectName("menubar")
        MplMainWindow.setMenuBar(self.menubar)
        self.statusbar = QtWidgets.QStatusBar(MplMainWindow)
        self.statusbar.setObjectName("statusbar")
        MplMainWindow.setStatusBar(self.statusbar)

        self.retranslateUi(MplMainWindow)
        QtCore.QMetaObject.connectSlotsByName(MplMainWindow)

    def retranslateUi(self, MplMainWindow):
        _translate = QtCore.QCoreApplication.translate
        MplMainWindow.setWindowTitle(_translate("MplMainWindow", "MainWindow"))
        self.groupBox.setTitle(_translate("MplMainWindow", "GroupBox"))
        self.buttonDrawDate.setText(_translate("MplMainWindow", "Draw"))
        self.buttonErase.setText(_translate("MplMainWindow", "Erase"))

from mplwidgettest import MplWidgetTest

mplwidgettest.py file that contains the method ‘setTema’ to update matplotlib’s style. When I print the variable rcParams it contains axes.facecolor: black but it does not apply

from PyQt5.QtWidgets import QSizePolicy, QWidget, QVBoxLayout
from matplotlib.figure import Figure
from matplotlib.backends.backend_qt5agg import (
    FigureCanvasQTAgg as FigureCanvas,
    NavigationToolbar2QT as NavigationToolbar)
import matplotlib as mplib


class MplCanvas(FigureCanvas):
    """Class to represent the FigureCanvas widget"""
    def __init__(self):
        # setup Matplotlib Figure and Axis
        mplib.rcParams.update(mplib.rcParamsDefault)
        mplib.style.use('bmh')
        self.fig = Figure()
        self.ax = self.fig.add_subplot(111)
        # initialization of the canvas
        FigureCanvas.__init__(self, self.fig)
        # we define the widget as expandable
        FigureCanvas.setSizePolicy(self,
                                   QSizePolicy.Expanding,
                                   QSizePolicy.Expanding)
        # notify the system of updated policy
        FigureCanvas.updateGeometry(self)

    def setTema(self, tema='classic'):
        print(tema)
        mplib.style.use(tema)
        print(mplib.rcParams)
        mplib.rcParams.update(mplib.rcParams)


class MplWidgetTest(QWidget):
    """Widget defined in Qt Designer"""
    def __init__(self, parent=None):
        # initialization of Qt MainWindow widget
        QWidget.__init__(self, parent)
        # set the canvas to the Matplotlib widget
        self.canvas = MplCanvas()
        # create a NavigatioToolbar
        self.ntb = NavigationToolbar(self.canvas, self)
        # create a vertical box layout
        self.vbl = QVBoxLayout()
        # add mpl widget to vertical box
        self.vbl.addWidget(self.canvas)
        # add NavigationToolBar to vertical box
        self.vbl.addWidget(self.ntb)
        # set the layout to th vertical box
        self.setLayout(self.vbl)

mainMplWidget.py File that calls the two previous files, and contains the method ‘cambiarTema’ that changes the matplotlib’s style

import sys
from IHMDrawDates import Ui_MplMainWindow
from PyQt5.QtWidgets import QMainWindow, QApplication
from PyQt5.QtGui import QCursor
import numpy as np


class DesignerMainWindow(QMainWindow, Ui_MplMainWindow):

    def __init__(self, parent=None):
        super(DesignerMainWindow, self).__init__(parent)
        self.setupUi(self)
        self.posX = []
        self.posY = []
        temas = ['bmh', 'classic', 'grayscale', 'seaborn-bright',
                 'ggplot', 'dark_background']
        self.comboTema.addItems(temas)
        # connect the signals with the slots
        self.buttonDrawDate.clicked.connect(self.drawDate)
        self.buttonErase.clicked.connect(self.eraseDate)
        self.comboTema.currentIndexChanged.connect(self.cambiarTema)

    def cambiarTema(self):
        tema = self.comboTema.currentText()
        self.mpl.canvas.setTema(str(tema))
        self.mpl.canvas.ax.clear()
        self.mpl.canvas.draw()
        self.mpl.canvas.flush_events()

    def drawDate(self):
        x = np.arange(0, 100, 0.1)
        y = np.sin(x)
        self.mpl.canvas.ax.plot(x, y)
        self.mpl.canvas.ax.relim()
        self.mpl.canvas.ax.autoscale(True)
        self.mpl.ntb.update()
        self.mpl.ntb.push_current()
        self.mpl.canvas.draw()

    def eraseDate(self):
        self.mpl.canvas.ax.clear()
        self.mpl.ntb.update()
        self.mpl.ntb.push_current()
        self.mpl.canvas.draw()

if __name__ == '__main__':
    app = 0
    app = QApplication(sys.argv)
    dmw = DesignerMainWindow()
    # show it
    dmw.show()
    sys.exit(app.exec_())


Get this bounty!!!

#StackBounty: #python #python-3.x #parsing #unit-testing #pandas Pytest fixture for testing a vertex-parsing function

Bounty: 50

I have just started using pytest and I am still getting used to how they do things. It seems like fixtures are at the core of the library, and that they can be used for making small pieces of dummy data that will get reused. I see that there are other methods for handling large dummy data. I have the following test code which tests a module I wrote called generate_kml.

import pytest
import generate_kml as gk
import pandas


@pytest.fixture
def line_record():
    return pandas.Series({gk.DB_VERTICES: "LINESTRING(1.1 1.1,2.2 2.2)"})


def test_convert_wkt_to_coords(line_record):
    expected = pandas.Series({gk.DB_VERTICES: [("1.1", "1.1"), ("2.2", "2.2")]})
    assert gk.convert_wkt_vertices_to_coords(line_record).equals(expected)

I am wondering if this is the way fixtures are meant to be used; to set up small reused data. (I plan to use the line_record multiple times in the test file). Additionally, I am wondering about the readability or redundancy of assigning the expected value to expected. If I directly compared the two Series, the line would exceed PEP8’s recommended line length, so I would break it into two lines anyway. If it adds readability here, then would it be good practice to always assign the expected value to a variable called expected (assuming you are comparing values that are expected to be equal)?
Here is the function being tested from generate_kml:

def convert_wkt_vertices_to_coords(vertices_as_wkt):
    def parse_coords(wkt):
        wkt = wkt[wkt.find("(") + 1:wkt.find(")")]
        coords = wkt.split(",")
        coords = [tuple(x.split(" ")) for x in coords]
        return coords
    return vertices_as_wkt.apply(parse_coords)

One more thing here; in convert_wkt_vertices_to_coords I have a nested function. I don’t plan on reusing it, but in the past I haven’t had a need for nested functions, so it feels a bit off to me. Should I leave it as a nested function or break it out as its own function in the module?


Get this bounty!!!

#StackBounty: #python #python-3.x #python-sphinx How can I use Sphinx with subpackages without duplicating everything?

Bounty: 50

I have the following package structure as a minimal example (for convenience, all is uploaded here):

.
├── sphinx
│   ├── build
│   ├── Makefile
│   └── source
│       ├── conf.py
│       ├── index.rst
│       └── train.rst
└── train
    ├── __init__.py
    └── train.py

When writing Python packages, one must specifiy the __all__ constant in the __init__.py of any package in order for Sphinx to be able to map a reference such as train.DatasetMeta to train.train.DatasetMeta or similar. However, sphinx-apidoc generates the following sections for these packages:

train package
=============

Submodules
----------

train.train module
------------------

.. automodule:: train.train
    :members:
    :undoc-members:
    :show-inheritance:


Module contents
---------------

.. automodule:: train
    :members:
    :undoc-members:
    :show-inheritance:

Which duplicates the entire documentation as it contains .. automodule:: module.file as well as .. automodule:: module, which refer to the same thing. Removing either of these sections results in undefined reference warnings (turned into errors when using -n to SPHINXOPTS).

sphinx_test/train/train.py:docstring of train.DatasetMeta:1:py:class reference target not found: train.train.DatasetMeta

How can I solve this?

train/train.py

from collections import namedtuple


class DatasetMeta(namedtuple('DatasetMeta', ['dataset', 'num_classes', 'shape'])):
    @property
    def size(self):
        '''int: Number of examples in the dataset'''
        return self.shape[0]

train/init.py

from .train import *

__all__ = ['DatasetMeta']

sphinx/source/conf.py

import os
import sys
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('../../'))


project = 'test'
copyright = ''
author = ''

version = ''
release = '0'

extensions = [
    'sphinx.ext.autodoc',
]

source_suffix = '.rst'
master_doc = 'index'

I just cannot figure out what the logic is here.


Get this bounty!!!

#StackBounty: #python-3.x #tensorflow #linear-regression #tensorboard Parameters shooting to infinity while training after some epochs

Bounty: 50

I’m implementing Linear Regression in Tensorflow first time. Initially,I tried it using linear model but after few iterations of training, my parameter shot up to infinity. So, I changed my model to a quadratic one and again tried training but still after few iterations of epochs, the same thing is happening.

Hence, the parameter in tf.summary.histogram(‘Weights’, W0) is receiving inf as parameter and similar is the case with W1 and b1.

I wanted to see my parameters in tensorboard(because I’ve never worked with it) but getting this error.

I have asked the question previously but the slight change was that I was using linear model which again was giving the same problem(I didn’t know that it was due to the parameters going to infinity because I was running this in my Ipython Notebook but when I ran the program in terminal, the below mentioned error was generated, which helped me figure out that the problem was due to the parameters shooting to infinity ). In the comments section, I got to know that it was working in someone’s PC, and his tensorboard showed that the parameters were actually reaching infinity.

Here is the link of the problem asked earlier.
I hope that I’ve correctly declared Y_ in my program else do correct me !

Here is the code in Tensorflow:

import tensorflow as tf
import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt

boston=load_boston()
type(boston)
boston.feature_names

bd=pd.DataFrame(data=boston.data,columns=boston.feature_names)

bd['Price']=pd.DataFrame(data=boston.target)
np.random.shuffle(bd.values)


W0=tf.Variable(0.3)
W1=tf.Variable(0.2)
b=tf.Variable(0.1)
#print(bd.shape[1])

tf.summary.histogram('Weights', W0)
tf.summary.histogram('Weights', W1)
tf.summary.histogram('Biases', b)



dataset_input=bd.iloc[:, 0 : bd.shape[1]-1];
#dataset_input.head(2)

dataset_output=bd.iloc[:, bd.shape[1]-1]
dataset_output=dataset_output.values
dataset_output=dataset_output.reshape((bd.shape[0],1)) 
#converted (506,) to (506,1) because in pandas
#the shape was not changing and it was needed later in feed_dict


dataset_input=dataset_input.values  #only dataset_input is in DataFrame form and converting it into np.ndarray


dataset_input = np.array(dataset_input, dtype=np.float32) 
#making the datatype into float32 for making it compatible with placeholders

dataset_output = np.array(dataset_output, dtype=np.float32)

X=tf.placeholder(tf.float32, shape=(None,bd.shape[1]-1))
Y=tf.placeholder(tf.float32, shape=(None,1))

Y_=W0*X*X + W1*X + b    #Hope this equation is rightly written
#Y_pred = tf.add(tf.multiply(tf.pow(X, pow_i), W), Y_pred)
print(X.shape)
print(Y.shape)


loss=tf.reduce_mean(tf.square(Y_-Y))
tf.summary.scalar('loss',loss)

optimizer=tf.train.GradientDescentOptimizer(0.001)
train=optimizer.minimize(loss)

init=tf.global_variables_initializer()#tf.global_variables_initializer()#tf.initialize_all_variables()
sess=tf.Session()
sess.run(init)



wb_=[]
with tf.Session() as sess:
    summary_merge = tf.summary.merge_all()

    writer=tf.summary.FileWriter("Users/ajay/Documents",sess.graph)

    epochs=10
    sess.run(init)

    for i in range(epochs):
        s_mer=sess.run(summary_merge,feed_dict={X: dataset_input, Y: dataset_output})  #ERROR________ERROR
        sess.run(train,feed_dict={X:dataset_input,Y:dataset_output})

        #CHANGED
        sess.run(loss, feed_dict={X:dataset_input,Y:dataset_output})
        writer.add_summary(s_mer,i)

        #tf.summary.histogram(name="loss",values=loss)
        if(i%5==0):
            print(i, sess.run([W0,W1,b]))
            wb_.append(sess.run([W0,W1,b]))

print(writer.get_logdir())
print(writer.close())

I’m getting this error :

 /anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
(?, 13)
(?, 1)
2018-07-22 02:04:24.826027: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
0 [-3833776.2, -7325.9595, -15.471448]
5 [inf, inf, inf]
Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Infinity in summary histogram for: Biases
     [[Node: Biases = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Biases/tag, Variable_2/read)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "LR.py", line 75, in <module>
    s_mer=sess.run(summary_merge,feed_dict={X: dataset_input, Y: dataset_output})  #ERROR________ERROR
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Infinity in summary histogram for: Biases
     [[Node: Biases = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Biases/tag, Variable_2/read)]]

Caused by op 'Biases', defined at:
  File "LR.py", line 24, in <module>
    tf.summary.histogram('Biases', b)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/summary/summary.py", line 187, in histogram
    tag=tag, values=values, name=scope)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 283, in histogram_summary
    "HistogramSummary", tag=tag, values=values, name=name)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Infinity in summary histogram for: Biases
     [[Node: Biases = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Biases/tag, Variable_2/read)]]


Get this bounty!!!

#StackBounty: #python #python-3.x #distutils2 Python 3.5 create .rpm with pinstaller generated executable

Bounty: 50

I’ve got a build generated with a pyinstaller.
I need to create .rpm package which will put the executable into the /usr/bin/ and create a systemd service which will run that executable.

I found this
https://docs.python.org/3/distutils/builtdist.html and https://docs.python.org/2.0/dist/creating-rpms.html

However it doesn’t give me a full picture.

  1. Is it possible to make it?

  2. What toolset do i need to use? (Basically, how to make it).

  3. If possible – sample code


Get this bounty!!!