#StackBounty: #python #tensorflow #tensorboard TensorBoard recording stats separate from training

Bounty: 50

I’m trying to use TensorBoard to display some graphs of a neural network training run. (That is, graphs of test and validation accuracy during training, not just of the network structure.) There is some example code

As well as some questions on this site, all of which seem to follow the same pattern as the example code. That is, the pattern always revolves around something like

summary, _ = sess.run([merged, train_step], ...

So basically, the operations of running a training step and recording statistics for graph display, are being conflated.

This is fine as far as it goes, but I’m trying to retrofit the graph to an existing program that inevitably does things in a slightly different way, so the example code won’t work as is. What I really want to do is isolate some code that just records the statistics, separate from existing code to do the training.

How do you record statistics for TensorBoard, within the main training loop, but separate from the code that does the training?


Get this bounty!!!

#StackBounty: #python #machine-learning #scikit-learn #jupyter #decision-tree Plot Interactive Decision Tree in Jupyter Notebook

Bounty: 50

Is there a way to plot a decision tree in a Jupyter Notebook, such that I can interactively explore its nodes? I am thinking about something like this dt. This is an example from KNIME.

I have found https://planspace.org/20151129-see_sklearn_trees_with_d3/ and https://bl.ocks.org/ajschumacher/65eda1df2b0dd2cf616f and I know you can run d3 in Jupyter, but I have not found any packages, that do that.


Get this bounty!!!

#StackBounty: #python #scikit-learn #anomaly-detection scikit-learn IsolationForest anomaly score

Bounty: 50

According to IsolationForest papers (refs are given in documentation)
the score produced by Isolation Forest should be between 0 and 1.

The implementation in scikit-learn negates the scores (so high score is more on inlier) and also seems to shift it by some amount. I’ve tried to figure out how to reverse it but was not successful so far. The code has some methods and attributes like score_samples() and self.offset_ that are not accessible from a fitted object. The documentation and comments in code on usage of self.contamination seem contradictory…

I have version 19.1 of scikit-learn (can’t tell if there were significant changes in IsolationForest implementation since then)

Any ideas/suggestions would be appreciated!


Get this bounty!!!

#StackBounty: #python #pdf #random-generation #kernel-smoothing #density-estimation Sample multivariate PDF from KDE with different norm

Bounty: 100

I am using KDE with a modified metric for the distance. The PDF is as expected (see below). But due to the new metric, I cannot use the usual sampling methods as they suppose a gaussian kernel with $(x – x_i)$.

  • How to generate new sample in this context?

From what I read (Find CDF from an estimated PDF (estimated by KDE), for instance), I have to come up with the multivariate inverse CDF.

  • Is there a simple way to do this?

For now I just use a hack which consists in:

  1. Sample the multivariate space and get PDF values
  2. Then I use these two information with a uniform random generator to give me a sample.

It works but will get the curse of dimensionality. Something like this: https://stackoverflow.com/q/25642599

Even the CDF in a multivariate space would be enough I guess as I would be able to use some fixed point for example to do the inverse.

Multivariate PDF with new distance metric


Get this bounty!!!

#StackBounty: #python #python-3.x #oop #monkeypatching Is it possible to fully Monkey Patch builtin `str`

Bounty: 50

I Am trying to patch python’s built-in str in order to track the count of all strallocations. I am running into some issues and was wondering if anyone could see what I’m doing wrong, or if this is even possible natively through monkey patching in python3?

$ python
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux

I first naively tried to patch str as if it were a function:

def patch_str_allocations():
  old_str = str
  def mystr(*args, **kwargs):
    return old_str(*args, **kwargs)

  builtins.str = mystr


def test():
  logger = logging.getLogger(__name__)
  patch_str_allocations()
  logger.debug(str('test'))

But of course this fails all sorts of operations that string is used for like isinstance

    logger.debug(route)
  File "/usr/lib/python3.5/logging/__init__.py", line 1267, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/usr/lib/python3.5/logging/__init__.py", line 1403, in _log
    fn, lno, func, sinfo = self.findCaller(stack_info)
  File "/usr/lib/python3.5/logging/__init__.py", line 1360, in findCaller
    filename = os.path.normcase(co.co_filename)
  File "/home/ubuntu/.virtualenvs/papm/lib/python3.5/posixpath.py", line 52, in normcase
    if not isinstance(s, (bytes, str)):
TypeError: isinstance() arg 2 must be a type or tuple of types

I then tried a class based approach:

class StrAllocator(str):
    oldstr = None

    def __new__(cls, *args, **kwargs):
        return StrAllocator.oldstr.__new__(cls, *args, **kwargs)

    @property
    def __class__(self):
        return str

    def __repr__(self):
        return self.__class__.__name__ + str.__repr__(self)


def patch_str_allocations():
    StrAllocator.oldstr = str
    builtins.str = StrAllocator

In normal str construction this is working OK but am still running into some issues:

class StrAllocatorTestCase(unittest.TestCase):

    def test_log(self):
        t1 = str('t1')
        logger = logging.getLogger(__name__)
        patch_str_allocations()
        t2 = str('t2')
        print(type(t1))
        print(type(t2))
        print(isinstance(t1, str))
        print(isinstance(t2, StrAllocator))
        print(isinstance(t2, str))
        logger.debug(str('test'))


$ nosetests tests.test_str_allocator:StrAllocatorTestCase.test_log -s

<class 'str'>
<class 'pythonapm.instruments.allocations.StrAllocator'>
False
True
True
E
======================================================================
ERROR: test_log (tests.instruments.test_str_allocator.StrAllocatorTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/vagrant_data/github.com/dm03514/python-apm/tests/instruments/test_str_allocator.py", line 30, in test_log
    logger.debug(str('test'))
  File "/usr/lib/python3.5/logging/__init__.py", line 1267, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/usr/lib/python3.5/logging/__init__.py", line 1403, in _log
    fn, lno, func, sinfo = self.findCaller(stack_info)
  File "/usr/lib/python3.5/logging/__init__.py", line 1360, in findCaller
    filename = os.path.normcase(co.co_filename)
  File "/home/ubuntu/.virtualenvs/papm/lib/python3.5/posixpath.py", line 54, in normcase
    "not '{}'".format(s.__class__.__name__))
TypeError: normcase() argument must be str or bytes, not 'str'

----------------------------------------------------------------------
Ran 1 test in 0.003s

Can anyone see what’s missing? (besides my understanding of descriptors and python classes :p )


From the REPL the example above Works, but it does not work within the context of nose and unittests…

⟫ ipython
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import logging

In [2]: import builtins

In [3]: class StrAllocator(str):
   ...:     oldstr = None
   ...:
   ...:     def __new__(cls, *args, **kwargs):
   ...:         return StrAllocator.oldstr.__new__(cls, *args, **kwargs)
   ...:
   ...:     @property
   ...:     def __class__(self):
   ...:         return str
   ...:
   ...:     def __repr__(self):
   ...:         return self.__class__.__name__ + str.__repr__(self)
   ...:
In [4]: def patch_str_allocations():                                                                                                                                    [6/9733]
   ...:     StrAllocator.oldstr = str
   ...:     builtins.str = StrAllocator
   ...:

In [5]:   def test_log():
   ...:         t1 = str('t1')
   ...:         logger = logging.getLogger(__name__)
   ...:         patch_str_allocations()
   ...:         t2 = str('t2')
   ...:         print(type(t1))
   ...:         print(type(t2))
   ...:         print(isinstance(t1, str))
   ...:         print(isinstance(t2, StrAllocator))
   ...:         print(isinstance(t2, str))
   ...:         logger.debug(str('test'))
   ...:
In [6]: test_log()
<class 'str'>
<class '__main__.StrAllocator'>
False
True
True


Get this bounty!!!

#StackBounty: #python #c #shellcode #segmentation #cross-platform Python – ctypes.cast function call outputs Segmentation Fault error m…

Bounty: 50

Background:

Writing a proof of concept of a twitter C&C inspired by Prof Viviek of SPSE similiar to twittor and the project includes a functionality of injecting shellcode and executing it within the python’s program. However I wanted to make the functionality cross platforms, but it only worked in a windows system.

Problem: When executing the python script in ubuntu server 12.04.5 LTS, the output says

Segmentation fault (core dumped)

Which means I am getting denied for accessing memory that I don’t have permission for. This is strange b/c in the source code I also set cytpes.mprotect(allocated_space, space_size, 7) <== 4 + 2 + 1 is for wrx permissions

Shellcode generated using command:

msfvenom --payload linux/x86/shell/bind_tcp  --format py --arch x86 --bad-char "x00x20x0d"

Python Script:

 #!/usr/bin/env python

import ctypes
import os

# please comment out the appropriate payload for the other platforms
# this below is for windows shell bind tcp listening at 4444
#buf  = ""
#buf += "shellcode..."

# below is for linux
buf = ""
buf += "shellcode..."


def main(shellcode):
    if os.name == 'posix':                                                 
        try:                              
            libc = ctypes.CDLL('libc.so.6')                                
            sc_ptr = ctypes.c_char_p(shellcode)                               

            size = len(shellcode)                                          
            addr_freespace = ctypes.c_void_p(libc.valloc(size))                 
            ctypes.memmove(addr_freespace, sc_ptr, size)                            
            libc.mprotect(addr_free_space, size, 1 | 2 | 4)   # changed to 7 for all three access                    
            run = ctypes.cast(free_space, ctypes.CFUNCTYPE(ctypes.c_void_p))
            run()                                                           
            sys.exit()                                                                    
        except Exception as e:
            print "Error: " e

    else:                                                                   
        try:  # windows implementation



if __name__ == '__main__':
    main(buf)

Question: Can someone explain why does the segmentation fault message appear and how do we fix such problem?

Credits: this script is inspired by sickle.py @ Line743-753

The only difference is the reference script is using python 3, while I am using python 2.7.

UPDATE:

After many trials and errors, including running the program in pdb. The segmentation fault error happened after the line of:

run()

Can someone please explain why this is happening?


Get this bounty!!!

#StackBounty: #python #tkinter #tkmessagebox How can I add a "show details" button to a tkinter messagebox?

Bounty: 50

I have a Python script which uses tkinter.messagebox to display an error message with traceback details if an unexpected exception occurs.

import tkinter.messagebox as tm
import traceback

try:
    1/0
except Exception as error:
    message = "An error has occurred: '" + str(error) + "'."
    detail = traceback.format_exc()
    tm.showerror(title="Error", message=message, detail=detail)

Standard tkinter error

Displaying tracebacks with the -detail attribute has a few drawbacks.

Instead of displaying error details by default, I would like to add a “show details” button which would display more information in a read-only text field.

Detailed error for "division by zero"

How can I add a “show details” button to a tkinter messagebox?


Get this bounty!!!

#StackBounty: #python #numpy #matplotlib Matplotlib canvas as numpy array artefacts

Bounty: 50

I want to convert a matplotlib figure into a numpy array. I have been able to do this by accessing the contents of the renderer directly. However, when I call imshow on the numpy array it has what looks like aliasing artefacts along the edges which aren’t present in the original figure.

I’ve tried playing around with various parameters but can’t figure out how to fix the artefacts from imshow. The differences in the images remain if I save the figures to an image file.

Note that what I want to achieve is a way to confirm that the content of the array is the same as the figure I viewed before. I think probably these artefacts are not present in the numpy array but are created during the imshow call. Perhaps approriate configuration of imshow can resolve the problem.

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.patches import Rectangle
import math

fig = plt.figure(frameon=False)
ax = plt.gca()
ax.add_patch(Rectangle((0,0), 1, 1, angle=45, color="red"))
ax.set_xlim(-2,2)
ax.set_ylim(-2,2)
ax.set_aspect(1)
plt.axis("off")
fig.canvas.draw()
plt.savefig("rec1.png")
plt.show()
X = np.array(fig.canvas.renderer._renderer)

fig = plt.figure(frameon=False)
ax = plt.gca()
plt.axis("off")
plt.imshow(X)
plt.savefig("rec2.png")
plt.show()

enter image description here


Get this bounty!!!

#StackBounty: #python #object-oriented #python-3.x #numpy #pandas Warhammer: How many of my attacks will succeed?

Bounty: 100

Me and a couple of mates sometimes play a game called Warhammer.
When playing the game you have options of what each model attacks.
This can lead to situations where you know if you shoot with 100% of your units into one enemy unit you know the unit will be killed, but you don’t know how many you should fire to kill the target unit.
And so I decided to write a little program that would help find out how many models I should shoot with into an enemy.

Combat in Warhammer is pretty basic, however some added complexity can come from additional rules on specific units or weapons.
The core rules when attacking another unit with a model is:

  1. Choose a Model to fight with
  2. Choose the Unit(s) to attack
  3. Choose the weapons you’ll attack with
  4. Resolve Attacks:
    1. Hit roll: for each attack roll a dice, if the roll is greater or equal to the attacking models Skill the attack hits.
    2. Wound roll: This is the same as hitting, however what you roll is based on the weapons Strength and the targets Toughness.
      • S >= 2T: 2+
      • S > T: 3+
      • S == T: 4+
      • S < T: 5+
      • S <= T: 6+
    3. Allocate wound: You select a model to try and resist the wound.

    4. Saving Throw: Roll a dice and add armor penetration to the roll, if it’s greater than the models save then no damage is inflicted.

      There are also ‘invulnerable saves’, which work the same way as normal saves, but aren’t affected by armor penetration.

    5. Inflict Damage: The model takes the weapons damage, if the unit is reduced to 0 wounds it dies.

An example of this is:

  1. We select a Khorne Berzerker

    $
    begin{array}{l|l|l|l|l}
    textrm{Skill} &
    textrm{S} &
    textrm{T} &
    textrm{W} &
    textrm{Sv} \
    hline
    text{3+} &
    text{5} &
    text{4} &
    text{1} &
    text{3+} \
    end{array}
    $

  2. We attack a squad of Khorne Berzerkers

  3. We will attack with it’s Chainaxe

    $
    begin{array}{l|l|l|l}
    textrm{Attacks} &
    textrm{S} &
    textrm{AP} &
    textrm{D} \
    hline
    text{1} &
    text{6} &
    text{-1} &
    text{1} \
    end{array}
    $

    1. I roll a 3. This is equal to the models Skill.
    2. I roll a 3. This is equal to the required roll. (6 > 4: 3+)
    3. A Khorne Berzerker is selected to take the wound.
    4. My opponent rolls a 3. And since $3 – 1 < 3$, the save is failed, and the wound goes through.
    5. One enemy model dies.

There are some additional common effects:

  • Some units allow others to re-roll failed hit rolls, hit rolls of one, failed wound rolls and wound rolls of one. However you can only re-roll a roll once, so you couldn’t re-roll a hit of 1 and then re-roll a hit of 2. But you can re-roll a failed hit and then re-roll a failed wound.
  • Some things allow you to add to your hit and wound rolls.
  • Some things allow you to skip your hit or wound phase. Flame throwers normally auto hit, and so skip their hit phase.

And so I wrote some code to show the percentage of attacks that will be lost, and at what stage.
And the average amount of attacks and damage each weapon will have.

from functools import wraps
import enum
from collections import Counter
from itertools import product

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np


class TypedProperty:
    def __init__(self, name, *types):
        types = [type(None) if t is None else t for t in types]
        if not all(isinstance(t, type) for t in types):
            raise ValueError('All arguments to `types` must inherit from type.')
        self.types = tuple(types)
        self.name = name

    def __get__(self, obj, _):
        return self._get(obj, self.name)

    def __set__(self, obj, value):
        if not isinstance(value, self.types):
            raise TypeError('Value {value} must inherit one of {self.types}'.format(value=value, self=self))
        self._set(obj, self.name, value)

    def __delete__(self, obj):
        self._delete(obj, self.name)

    def get(self, fn):
        self._get = fn
        return self

    def set(self, fn):
        self._set = fn
        return self

    def delete(self, fn):
        self._delete = fn
        return self

    @staticmethod
    def _get(self, name):
        return getattr(self, name)

    @staticmethod
    def _set(self, name, value):
        setattr(self, name, value)

    @staticmethod
    def _delete(self, name):
        delattr(self, name)


class Damage(tuple):
    def __new__(self, value):
        if isinstance(value, tuple):
            pass
        elif isinstance(value, int):
            value = (value, None)
        elif not isinstance(value, str):
            raise TypeError('Value must be an int, tuple or str')
        else:
            value = tuple(value.split('d', 1) + [None])[:2]
            value = (i or None for i in value)

        value = tuple(int(i) if i is not None else 1 for i in value)
        return super().__new__(self, value)


class Effects(enum.Enum):
    SKIP_HIT = 0
    HIT_ONE = 1
    HIT_FAILED = 2
    WOUND_ONE = 3
    WOUND_FAILED = 4


class Base:
    _INIT = tuple()

    def __init__(self, *args, **kwargs):
        values = self._read_args(args, kwargs)
        for name in self._INIT:
            setattr(self, name, values.get(name, None))

    def _read_args(self, args, kwargs):
        values = dict(zip(self._INIT, args))
        values.update(kwargs)
        return values


class User(Base):
    _INIT=tuple('skill'.split())
    skill=TypedProperty('_skill', int)


class Weapon(Base):
    _INIT=tuple('attacks strength ap damage'.split())
    attacks=TypedProperty('_attacks', Damage)
    strength=TypedProperty('_strength', int)
    ap=TypedProperty('_ap', int)
    damage=TypedProperty('_damage', Damage)


class Target(Base):
    _INIT=tuple('toughness save invulnerable'.split())
    toughness=TypedProperty('_toughness', int)
    save=TypedProperty('_save', int)
    invulnerable=TypedProperty('_invulnerable', int, None)


class RoundEffects(Base):
    _INIT=tuple('skip one failed increase'.split())
    skip=TypedProperty('_skip', bool, None)
    one=TypedProperty('_one', bool, None)
    failed=TypedProperty('_failed', bool, None)
    increase=TypedProperty('_increase', int, None)

    def reroll(self, score):
        if self.failed:
            return score
        if self.one:
            return 1
        return 0

    def round(self, score):
        if self.skip:
            return None
        return (
            score + (self.increase or 0),
            self.reroll(score)
        )


class Effects(Base):
    _INIT=tuple('hit wound'.split())
    hit=TypedProperty('_hit', RoundEffects)
    wound=TypedProperty('_wound', RoundEffects)

    def __init__(self, *args, **kwargs):
        kwargs = self._read_args(args, kwargs)
        for key in 'hit wound'.split():
            if kwargs.get(key, None) is None:
                kwargs[key] = RoundEffects()
        super().__init__(**kwargs)


class Instance(Base):
    _INIT=tuple('user weapon target effects'.split())
    user=TypedProperty('_user', User)
    weapon=TypedProperty('_weapon', Weapon)
    target=TypedProperty('_target', Target)
    effects=TypedProperty('_effects', Effects)

    def __init__(self, *args, **kwargs):
        kwargs = self._read_args(args, kwargs)
        if kwargs.get('effects', None) is None:
            kwargs['effects'] = Effects()
        super().__init__(**kwargs)

    def _damage(self, damage):
        amount, variable = damage
        variable = tuple(range(1, variable+1))
        return [sum(ns) for ns in product(variable, repeat=amount)]

    def attacks(self):
        return self._damage(self.weapon.attacks)

    def shots(self):
        return self.weapon.attacks

    def hits(self):
        return self.effects.hit.round(self.user.skill)

    def _round(self, damage):
        if damage is None:
            return (0, 100)
        needed, reroll = damage
        values = tuple(range(6))
        rolls = np.array([
            v
            for n in values
            for v in (values if n < reroll else [n] * 6)
        ])
        ratio = np.bincount(rolls >= needed)
        return ratio * 100 / np.sum(ratio)

    def hits_wl(self):
        return self._round(self.hits())

    def damage_roll(self):
        s = self.weapon.strength
        t = self.target.toughness
        if s >= t * 2:
            return 2
        if s > t:
            return 3
        if s == t:
            return 4
        if s * 2 <= t:
            return 6
        if s < t:
            return 5

    def wounds(self):
        return self.effects.wound.round(self.damage_roll())

    def wounds_wl(self):
        return self._round(self.wounds())

    def save(self):
        return min(
            self.target.save - self.weapon.ap,
            self.target.invulnerable or 7
        )

    def save_wl(self):
        save = self.save()
        ratio = np.array((7 - save, save - 1))
        return ratio * 100 / np.sum(ratio)

    def win_loss(self):
        wls = [
            self.hits_wl(),
            self.wounds_wl(),
            self.save_wl()
        ]
        failed = 0
        for loss, _ in wls:
            win = 100 - failed
            loss = loss * win / 100
            yield loss
            failed += loss
        yield 100 - failed

    def damage(self):
        return self._damage(self.weapon.damage)


def plot(instance):
    fig, axes = plt.subplots(1, 3)

    win_loss = list(instance.win_loss())
    df = pd.DataFrame(
        [
            win_loss[:1] + [0, 0] + [sum(win_loss[1:])],
            win_loss[:2] + [0] + [sum(win_loss[2:])],
            win_loss
        ],
        columns=['Miss', 'Prevented', 'Saved', 'Passed'],
        index=['Hit', 'Wound', 'Save']
    )
    df.plot.bar(stacked=True, ax=axes[1]).set_ylim(0, 100)

    attacks = instance.attacks()
    damage = instance.damage()
    limit = max(max(attacks), max(damage))
    limit = int((limit + 1) * 1.1)

    pd.DataFrame(attacks).boxplot(return_type='axes', ax=axes[0]).set_ylim(0, limit)
    pd.DataFrame(damage).boxplot(return_type='axes', ax=axes[2]).set_ylim(0, limit)



if __name__ == '__main__':
    khorn = Instance(
        User(skill=3),
        Weapon(
            attacks=Damage(2),
            strength=6,
            ap=-1,
            damage=Damage(1)
        ),
        Target(
            toughness=4,
            save=3
        ),
        Effects(
            RoundEffects(
                failed=True
            ),
            RoundEffects(
                failed=True
            )
        )
    )
    plot(khorn)

    khorn2 = Instance(
        User(skill=3),
        Weapon(
            attacks=Damage(2),
            strength=6,
            ap=-1,
            damage=Damage(1)
        ),
        Target(
            toughness=4,
            save=3
        )
    )
    plot(khorn2)

    land = Instance(
        User(skill=3),
        Weapon(
            attacks=Damage(2),
            strength=9,
            ap=-3,
            damage=Damage('d6')
        ),
        Target(
            toughness=7,
            save=3
        )
    )
    plot(land)

    predator = Instance(
        User(skill=3),
        Weapon(
            attacks=Damage('2d3'),
            strength=7,
            ap=-1,
            damage=Damage('3')
        ),
        Target(
            toughness=7,
            save=3
        )
    )
    plot(predator)

    plt.show()


Get this bounty!!!