#StackBounty: #python #visualization #beginner How to more simply see overlapping data for dozens of variables?

Bounty: 50

I’m trying to think of the best way to see how multiple variables (about 40) related to a very large userbase can be seen to interact with one another.

As an analogy, imagine I had survey data of superheros liked by thousands of school children, such that Adam Agner only likes Batman, Brian Agosto likes Superman and Batman, Cian Ailiff likes Wonderwoman, Superman and the Flash, etc. The index would be the list of children’s names (tens of thousands), and the variables would be each superhero (from a list of 50) each with a True or False value.

Rather than just the amount of children who like each superhero, if I can see the overlap information in some easy way, I might find that an unusually large number only like Batman, or that most of those who like Superman are likely to also like the Flash.

The easiest would be to do it visually, but Venn diagrams wouldn’t be practical with a large number (50 here) of variables. So it would seem like a correlation matrix with a heat-map base would be the way to go, with something like this:

Source: http://www.sthda.com/english/wiki/correlation-matrix-a-quick-start-guide-to-analyze-format-and-visualize-a-correlation-matrix-using-r-software

I can imagine a heat map with superheros plotted on both axes could work for seeing interesting matches of one variable against all other ones, but it would still require some less ideal extra steps to see when three or more superheros matched, like seeing the colour overlap of superhero1 with superhero2 & superhero3 both be high, and that’s kind of guesswork, because the actual children who like all 3 could still be low.

At the moment, the best solution I can think of is to try to reduce my variables down into categories, so the example could be Marvel, DC, male, female, but that loses some potential data that could be helpful from overlaps within those categories.

Maybe if I did something like the image above, the circle size would be the number for that overlap, and the colour could be number matches with other variables (without listing them, but I could further investigate). That kind of coding is a little out of my comfort zone, but I could try.

Ideas appreciated! I’m most experienced with Python & pandas (it’s how I merge the data), but if I have to use Matlab or another tool then I will consider that. Ideas or suggestions appreciated! Hopefully this was clear, thank you!


Get this bounty!!!

#StackBounty: #python #machine-learning #tensorflow #neural-network #keras Get audiences insights using Keras and TensorFlow

Bounty: 50

Recently I’ve discovered Keras and TensorFlow and I’m trying to get into ML. I have manually classified train and test data from my users DB like so:

9 features and a label, the features are events in my system like “user added a profile picture” or “user paid X for a service” and the label is positive or negative R.O.I (1 or 0)

Sample:
enter image description here

I have used the following code to classify the users:

import numpy as np
from keras.layers import Dense
from keras.models import Sequential

train_data = np.loadtxt("train.csv", delimiter=",", skiprows=1)
test_data = np.loadtxt("test.csv", delimiter=",", skiprows=1)

X_train = train_data[:, 0:9]
Y_train = train_data[:, 9]

X_test = test_data[:, 0:9]
Y_test = test_data[:, 9]

model = Sequential()
model.add(Dense(8, input_dim=9, activation='relu'))
model.add(Dense(6, activation='relu'))
model.add(Dense(3, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Fit the model
model.fit(X_train, Y_train, epochs=12000, batch_size=10)

# evaluate the model
scores = model.evaluate(X_test, Y_test)
print("nnnResults: %s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

And got a 89% accuracy. That worked great in order to label a user as a valued customer.

Q : How can I extract the features that contributed for the possitive R.O.I so I can boost their focus in the UX?

Or : What is the approach to find the best combined segment of audiences?


Get this bounty!!!

#StackBounty: #python #object-oriented #python-3.x #inheritance #optional Implementing a Maybe for multidimensional data

Bounty: 300

The scenario is we’re working with a REST endpoint that gives us nice JSON objects. We’re using requests, and everything works wonderfully. But one day you notice that data you though was always being pulled back isn’t there for some models! You’re getting index, key and attribute errors left, right and centre. What used to be obj.phones[0] is now:

(DefaultAttr(obj, None).phones or [None])[0]

Or worse, if obj.phones[0] could actually be None!

And so I decided to build a simple Maybe like group of classes. The interface in Haskell seemed really clean and simple. However there’s no case statement in Python, and not one that’s based on type. so I decided to use Maybe.get(default: T) -> T and Just._value to get the value instead. Since Just._value isn’t implemented on either Nothing or Maybe, I decided to make the equality check if the LHS is an instance of the RHS, and so Just(1) == Just would be True.

This however isn’t good enough for the JSON object, and so I subclassed Maybe to create a MaybeNav, that creates a MaybeNav when you get items or attributes. And so if you don’t care if obj.phones[0] is there or None, you can use MaybeNav(obj).phones[0].get(). Which is much cleaner, and simpler to read.

I’ve added type hints to the code, so that it’s a little simpler to reason with. And I’ve added some docstrings, I’d appreciate any way to improve these, as I don’t normally use them. And so my skills with them are likely to be very poor.

Due to using f-strings, the code only works in Python 3.6. Finally any and all help is welcome.

mayby.py

from typing import TypeVar, Generic

T = TypeVar('T')
class Maybe(Generic[T]):
    """Simple Maybe abstract base class"""
    def __init__(self, *args, **kwargs):
        """
        Error as this should be overwridden

        The code for this interface is implemented in Just and Nothing.
        This is to prevent implementation errors. Such as overwritting
        __getattr__ in a child class causing an infinate loop.
        """
        raise TypeError("'Maybe' needs to be instansiated by child constructor")

    def get(self, default: T = None) -> T:
        """Get the value in Just, or return default if Nothing"""
        raise NotImplementedError("'get' should be changed in a sublcass")


class Constant(type):
    def __call__(self, *args, **kwargs):
        """Simple metaclass to create a constant class"""
        # Constants can't be called.
        raise TypeError(f"'{self.__name__}' object is not callable")

    def __repr__(self) -> str:
        # Display the constant, rather than the class location in memory.
        return f'{self.__name__}'


class JustBase:
    def __init__(self, value: T):
        """Younger sibling class of Maybe for Just classes"""
        self.__value = value

    def __repr__(self) -> str:
        return f'{type(self).__name__}({self._value!r})'

    def __eq__(self, other: object) -> bool:
        """
        Check if this is an instance of other

        This makes checking if the class is a Just simpler.
        As it's a common operation.
        """
        return isinstance(other, type) and isinstance(self, other)

    def get(self, default: T = None) -> T:
        """Get the value in Just, or return default if Nothing"""
        return self._value

    @property
    def _value(self):
        return self.__value


def build_maybes(Maybe, just_name, nothing_name):
    """Build a Just and Nothing inheriting from Maybe"""
    Just = type(just_name, (JustBase, Maybe), {})
    class MaybeConstant(Constant, Maybe):
        def get(self, default: T = None) -> T:
            """Get the value in Just, or return default if Nothing"""
            return default
    Nothing = MaybeConstant(nothing_name, (object,), {})
    return Just, Nothing


class MaybeNav(Maybe[T]):
    """Maybe for navigating objects"""
    # __getitem__ and __getattr__ actually return MaybeNav[T].
    def __getitem__(self, item: object) -> Maybe[T]:
        if self == NothingNav:
            return NothingNav
        val = self._value
        try:
            val = val[item]
        except Exception:
            return NothingNav
        else:
            return JustNav(val)

    def __getattr__(self, item: str) -> Maybe[T]:
        obj = object()
        val = getattr(self.get(obj), item, obj)
        if val is obj:
            return NothingNav
        return JustNav(val)


Just, Nothing = build_maybes(Maybe, 'Just', 'Nothing')
JustNav, NothingNav = build_maybes(MaybeNav, 'JustNav', 'NothingNav')

# Don't delete T, so subclasses can use the same generic type
del TypeVar, Generic, Constant, JustBase

An example of using this can be:

import math
from collections import namedtuple
from typing import Iterable

from maybe import Maybe, Just, Nothing, MaybeNav, JustNav

def safe_log(number: float) -> Maybe[float]:
    if number > 0:
        return Just(math.log(number))
    else:
        return Nothing


def user_values(obj: object) -> Iterable[MaybeNav[object]]:
    obj = JustNav(obj)
    return [
        obj.name,
        obj.first_name,
        obj.last_name,
        obj.phones[0]
    ]


v = safe_log(1000)
print(v == Just, v)
v = safe_log(-1000)
print(v == Just, v)

User = namedtuple('User', 'name first_name last_name phones')
vals = user_values(User('Peilonrayz', 'Peilonrayz', None, []))
print(vals)
print([val.get(None) for val in vals])

I find maybe.py a little hard to read, mostly as I got the inheritance to happen in build_maybes. And so the following is a class diagram of the user defined objects in maybe.py. One thing to note is the dotted arrows from Nothing and NothingNav to MaybeConstant are to show they are using a metaclass, rather than box standard inheritance. The box with some classes in it is to emphasise they are created in build_maybes.

Class diagram


Get this bounty!!!

#StackBounty: #python #rabbitmq #celery start celery worker and enable it for broadcast queue

Bounty: 50

I’m trying to start celery worker so it only listens to single queue. This is not a problem, I can do this that way:

python -m celery worker -A my_module -Q my_queue -c 1

But now I also want this my_queue queue to be a broadcast queue, so I do this in my celeryconfig:

from kombu.common import Broadcast
CELERY_QUEUES = (Broadcast('my_queue'),)

But as soon as I do this I cannot start my worker anymore, I get error from rabbitmq:

amqp.exceptions.PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'my_queue' in vhost 'myvhost': received 'fanout' but current is 'direct'

If I start worker without -Q (but leaving Broadcast in celeryconfig.py as described above) and I list rabbitmq queues I can see broadcast queue is created and named like this:

bcast.43fecba7-786a-461c-a322-620039b29b8b

And similarly if I define this queue within worker (using -Q as mentioned above) or as simple Queue in celeryconfig.py like this:

from kombu import Queue
CELERY_QUEUES = (Queue('my_queue'),)

I can see this queue in rabbitmq like this:

my_queue

It apperas it does not matter what I put into Broadcast call when defining the queue – this seems to be internal celery name, not passed to rabbitmq.

So I’m guessing when worker is starting then my_queue is created and once that’s done it cannot be made Broadcast.

I can have a worker that listens to any queue (not only to my_queue) which I would start by removing the -Q argument. But it would be nice to be able to have a single process that only listens to that particular queue since my tasks I throw in there are fast and I’d like to bring latency down as much as possible.

— Edit 1 —
Spent some time with this problem and it seems bcast queue mentioned above does not appear consistently. After reseting rabbitmq and running celery without -Q option bcast queue did not appear…


Get this bounty!!!

#StackBounty: #python #sockets #ssl Python/sockets/ssl EOF occurred in violation of protocol

Bounty: 150

I would like to authenticate the server at client’s side in my echo client/server program. I’m using python 2.7.12 and the ssl module on

Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:        14.04
Codename:       trusty

I’ve generated client’s and server’s certificates and keys using the openssl commands:

openssl req -new -x509 -days 365 -nodes -out client.pem -keyout client.key
openssl req -new -x509 -days 365 -nodes -out server.pem -keyout server.key

Versions of openssl library itself and openssl used by python are the same:

openssl version -a
OpenSSL 1.0.1f 6 Jan 2014
built on: Fri Sep 23 12:19:57 UTC 2016
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) blowfish(idx) 
compiler: cc -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wa,--noexecstack -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/usr/lib/ssl"

python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 1.0.1f 6 Jan 2014

However, the code below shows some errors, at server’s side: EOF occurred in violation of protocol (_ssl.c:1645) (but the server still works), and at client’s side:

Traceback (most recent call last):
  File "/http_ssl_client.py", line 36, in <module>
    if not cert or ('commonName', 'test') not in cert['subject'][4]: raise Exception("Invalid SSL cert for host %s. Check if this is a man-in-themiddle attack!" )
Exception: Invalid SSL cert for host %s. Check if this is a man-in-themiddle attack!
{'notBefore': u'Jun  3 11:54:21 2017 GMT', 'serialNumber': u'BBDCBEED69655B6E', 'notAfter': 'Jun  3 11:54:21 2018 GMT', 'version': 3L, 'subject': ((('countryName', u'pl'),), (('stateOrProvinceName', u'test'),), (('localityName', u'test'),), (('organizationName', u'test'),), (('organizationalUnitName', u'test'),), (('commonName', u'test'),), (('emailAddress', u'test'),)), 'issuer': ((('countryName', u'pl'),), (('stateOrProvinceName', u'test'),), (('localityName', u'test'),), (('organizationName', u'test'),), (('organizationalUnitName', u'test'),), (('commonName', u'test'),), (('emailAddress', u'test'),))}

Server’s code:

#!/bin/usr/env python
import socket
import ssl

def main():
    HOST = '127.0.0.1'
    PORT = 1234

    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((HOST, PORT))
    sock.listen(5)

    while True:
        conn = None
        client_sock, addr = sock.accept()
        try:
            ssl_client = ssl.wrap_socket(client_sock, server_side=True, certfile="server.pem", keyfile="server.key", ssl_version=ssl.PROTOCOL_TLSv1_2)
            data =  ssl_client.read(1024)
            print data
            ssl_client.write(data)
        except ssl.SSLError as e:
            print(e)
        finally:
            if conn:
                conn.close()
if __name__ == '__main__':
    main()

Client:

#!/bin/usr/env python
import socket
import ssl

if __name__ == '__main__':

    HOST = '127.0.0.1'
    PORT = 1234

    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((HOST, PORT))

    context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
    context.verify_mode = ssl.CERT_REQUIRED
    context.load_verify_locations('server.pem')

    if ssl.HAS_SNI:
        secure_sock = context.wrap_socket(sock, server_hostname=HOST)
    else:
        secure_sock = context.wrap_socket(sock)

    cert = secure_sock.getpeercert()
    print cert

    if not cert or ('commonName', 'test') not in cert['subject'][4]: raise Exception("Error" )

    secure_sock.write('hello')

    print secure_sock.read(1024)

    secure_sock.close()
    sock.close()

All files are in the same directory.


Get this bounty!!!

#StackBounty: #python #python-2.7 #numpy #int #long-integer Why does int(maxint) give a long, but int(int(maxint)) give an int? Is this…

Bounty: 50

Pretty self-explanatory (I’m on Windows):

>>> import sys, numpy
>>> a = numpy.int_(sys.maxint)
>>> int(a).__class__
<type 'long'>
>>> int(int(a)).__class__
<type 'int'>

Why does calling int once give me a long, whereas calling it twice gives me an int?

Is this a bug or a feature?


Get this bounty!!!

#StackBounty: #python #youtube-dl Python – youtube-dl force login everytime

Bounty: 50

I want to download multiple files using youtube-dl from a site the needs login.

The issue that I have is that youtube-dl is login for the first video with no issue, but doesn’t login again for next video.

How do I force youtube-dl to login for each video, every time the function is called ? Maybe reset,stop youtube-dl every time.

def video_download(path, url):

        ydl = youtube_dl.YoutubeDL(
            {
                'outtmpl': path + '.mp4',
                'format': 'bestvideo+bestaudio/best',
                'username': email,
                'password': password,
                # 'quiet': True

            })

        with ydl:
            ydl.download([url])
        time.sleep(45)

The function is actually called in a loop, because I set outtmpl for each file/video.

Different instances, the first one is working, the next don’t login again.
I need every time the login to repeat.

<youtube_dl.YoutubeDL.YoutubeDL object at 0x0000000004E74C18>
<youtube_dl.YoutubeDL.YoutubeDL object at 0x00000000032BED68>
WARNING: Unable to download kaltura session JSON: HTTP Error 401: UNAUTHORIZED
<youtube_dl.YoutubeDL.YoutubeDL object at 0x0000000004D6D898>
WARNING: Unable to download kaltura session JSON: HTTP Error 401: UNAUTHORIZED

Terminal/Output:

<youtube_dl.YoutubeDL.YoutubeDL object at 0x00000000050E4C18>
**[safari] Downloading login form
[safari] Login successful**
[safari] 9781787283664/video1_1: Downloading webpage
.............................
<youtube_dl.YoutubeDL.YoutubeDL object at 0x000000000337DD68>
[safari] 9781787283664/video1_2: Downloading webpage
[safari] 9781787283664/video1_2: Downloading kaltura session JSON
WARNING: Unable to download kaltura session JSON: HTTP Error 401: UNAUTHORIZED


Get this bounty!!!

#StackBounty: #python Language/Library to use for unsafe/untrusted code hosted by python

Bounty: 50

I’m looking for a python3 library (or project) that can allow me to run “unsafe” code (input from the internet, or similar) without compromising security.

If such a library doesn’t exist, a recommendation for a programming language that I could “bind” (I’m not sure of the exact term, “host” maybe?) would also be acceptable (though I would like the solution to be as simple as possible).

For clarity, I’m not talking about “hosting” unsafe python code in python, which a few questions I have found here and on SO tell me is impossible. I would also like to avoid creating a DSL, because I want to support a lot of the features of a “modern” high-level programming language (algebraic expressions, list manipulation, class/function definition).

My specific requirements (roughly in order of importance):

  • Ability to pass “complex” data structures between the python host and the unsafe code. Ideally as much complexity as (for example) JSON would be supported, although I could probably manage with a bit less. Only being able to pass basic data types (int, string) would not work well.

  • The language the unsafe code is written in providing support for the language features mentioned above

  • Ability to call functions defined by the unsafe code from python (by name)

  • (Less important) ability for the unsafe code to call a function defined in the python code (safely)

  • (Optional) language for the unsafe code being simple and easy to learn

Some of the things that I’ve looked at are:

  • Python: as mentioned, it seems (nearly?) impossible to safely run unknown/untrusted python code. But I’ve seen some mentions that PyPy offers a restricted sandbox, or the RestrictedPython project. I’m not sure if these are considered safe or if they work for python3.
  • Javascript: via a library like PyV8 or PyExecJs. I haven’t found anything about whether running code this way is actually “safe”, but this seems like it could be a good solution. I’m also not sure if there’s a better maintained equivalent to these.

An answer that what I’m looking for doesn’t exist for python is also reasonable (though a pointer to a language/library solution where this is possible would be nice, in that case).


Get this bounty!!!

#StackBounty: #python #fill-paragraph Incorrect filling of Python docstrings in pep257 mode

Bounty: 50

I have python-fill-docstring-style set to pep-257 (default) and I’m trying to fill docstring paragraphs. However, python mode does not count """ towards the number of characters, nor does it consider indentation, resulting in:

def function(self):
    """a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
    a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
    a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
    """

and

class A:
    def function(self):
        """a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
        a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
        a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
        """

Note the incorrectly filled first row. How can I fix this?


Get this bounty!!!

#StackBounty: #python #design-patterns #recursion #backtracking #visitor-pattern A KenKen puzzle/solver in Python

Bounty: 200

I’ve written a simple KenKen puzzle/solver in Python. I’d love some feedback on the design of the puzzle, as well as the architecture for the solver.

To model the puzzle, I have the following classes:

  • Cell is used to model a (row, col, value) tuple
  • Cage (abstract) is used to model a grouping of Cell objects that must collectively satisfy a constraint. From this class, we have the following derived classes:
    • AddCage for cells involved in addition constraints
    • MulCage for cells involved in multiplication constraints
    • SubCage for cells involved in subtraction constraints
    • DivCage for cells involved in division constraints
    • ConCage for constant constraints
    • RowCage for unique row/column constraints
  • Puzzle combines cages, cells, and exposes methods for the unassigned cells, whether or not the puzzle is solved, etc.

Now for the code:

from abc import ABC, abstractmethod
from utils import kk_add, kk_mul, kk_sub, kk_div


class Cell:
    def __init__(self, row, col, value=None):
        """
        Models a cell in a kenken puzzle

        Args:
            row: row
            col: column
            value: cell value

        """

        self.row = row
        self.col = col
        self.value = value

    def __str__(self):
        return '<Cell ({0}, {1}): {2}>'.format(self.row, self.col, self.value)

    def __hash__(self):
        return hash((self.row, self.col))

    def accept(self, visitor):
        """
        Visitor implementation; accept a visitor object
        and call the object's visit method for this object

        Args:
            visitor: `CellVisitor` implementation 

        Returns: None
        """
        visitor.visit_cell(self)


class Cage(ABC):
    def __init__(self, cells, func):
        """
        Base class to model a cage in a kenken puzzle

        A cage is a grouping of cells with a constraint
        that the values of the cells must collectively satisfy

        Args:
            cells: the `Cell` objects in this cage
            func: a predicate used to indicate when the cage is satisfied

        """

        self.cells = set(cells)
        self.func = func

    def __str__(self):
        return '<{0} cells={1}>'.format(self.__class__.__name__, self.cells)

    @property
    def values(self):
        """ 
        Returns: list the cell values list for this cage
        """
        return [cell.value for cell in self.cells]

    @property
    def consistent(self):
        """
        Returns: bool whether or not this cage is consistent
        with respect to its current cell values
        """
        return None in self.values or self.solved

    @property
    def solved(self):
        """
        Returns: bool whether or not this cage is solved
        with respect to its current cell values
        """

        values = self.values
        return (
            None not in values
            and len(values) == len(self.cells)
            and self.evaluate(*values)
        )

    def evaluate(self, *values):
        """
        Evaluate this cage for the given input arguments,
        returning whether or not it's conditions have been satisfied

        Args:
            *values: variate list of test values

        Returns: bool
        """
        return self.func(values)

    @abstractmethod
    def accept(self, visitor):
        """
        Visit this cage. Accept a visitor object and call the
        object's visit method for this object

        Args:
            visitor: `CageVisitor` implementation 

        Returns: None
        """
        pass


class AddCage(Cage):
    def __init__(self, cells, value):
        """
        Models an addition cage in a kenken puzzle

        Args:
            cells: list of `Cell` objects contained in this cage
            value: target value the cell values in this cage must sum to

        """

        self.value = value
        super().__init__(cells, lambda values: kk_add(values, value))

    def accept(self, visitor):
        """
        Visit this cage

        Args:
            visitor: `CageVisitor` object

        Returns: None
        """
        visitor.visit_add(self)


class MulCage(Cage):
    def __init__(self, cells, value):
        """
        Models a multiplication cage in a kenken puzzle

        Args:
            cells: list of `Cell` objects contained in this cage
            value: target value the cell values in this cage must multiply to

        """

        self.value = value
        super().__init__(cells, lambda values: kk_mul(values, value))

    def accept(self, visitor):
        """
        Visit this cage

        Args:
            visitor: `CageVisitor` object

        Returns: None
        """
        visitor.visit_mul(self)


class SubCage(Cage):
    def __init__(self, cells, value):
        """
        Models a subtraction cage in a kenken puzzle

        Args:
            cells: list of `Cell` objects contained in this cage
            value: target value the cell values in this cage must subtract to

        """

        self.value = value
        super().__init__(cells, lambda values: kk_sub(values, value))

    def accept(self, visitor):
        """
        Visit this cage

        Args:
            visitor: `CageVisitor` object

        Returns: None
        """
        visitor.visit_sub(self)


class DivCage(Cage):
    def __init__(self, cells, value):
        """
        Models a division cage in a kenken puzzle

        Args:
            cells: list of `Cell` objects contained in this cage
            value: target value the cell values in this cage must divide to

        """

        self.value = value
        super().__init__(cells, lambda values: kk_div(values, value))

    def accept(self, visitor):
        """
        Visit this cage

        Args:
            visitor: `CageVisitor` object

        Returns: None
        """
        visitor.visit_div(self)


class ConCage(Cage):
    def __init__(self, cell, value):
        """
        Models a constant cage in a kenken puzzle

        Args:
            cell: `Cell` object for this cage
            value: target value

        """

        def func(values):
            return len(values) == 1 and values[0] == value

        self.value = value
        super().__init__([cell], func)

    def accept(self, visitor):
        """
        Visit this cage

        Args:
            visitor: `CageVisitor` object

        Returns: None
        """
        visitor.visit_con(self)


class RowCage(Cage): # RowConstraint
    def __init__(self, cells):
        """
        Models a row constraint in a kenken puzzle

        Here the cell values in this cage must be all unique
        for the cage to be solved

        Args:
            cells: `Cell` objects

        """

        def func(values):
            return len(values) == len(set(values))

        super().__init__(cells, func)

    def accept(self, visitor):
        """
        Visit this cage

        Args:
            visitor: `CageVisitor` object

        Returns: None
        """
        visitor.visit_row(self)


class Puzzle:
    def __init__(self, width, cells, cages):
        """
        Models a kenken puzzle

        See https://en.wikipedia.org/wiki/KenKen
        for more information

        Args:
            width: puzzle size
            cells: `Cell` objects comprising this puzzle
            cages: `Cage` objects a solution for this puzzle must satisfy

        """

        self.width = width
        self.cells = cells
        self.cages = cages

    def __str__(self):
        return '<Puzzle width={0}, cages={1}>'.format(
            self.width, len(self.cages)
        )

    @property
    def domain(self):
        """
        Returns: bool this puzzle's possible cells values
        """
        return range(1, self.width + 1)

    @property
    def unassigned(self):
        """
        Returns: bool this puzzle's unassigned cells
        """
        return (cell for cell in self.cells if cell.value is None)

    @property
    def solved(self):
        """
        Returns: bool whether or not this puzzle has been solved
        """
        return all(cage.solved for cage in self.cages)

    def consistent(self, cell):
        """
        Returns whether or not the value for the given cell is consistent
        with all of its cage constraints

        Args:
            cell: `Cell` object

        Returns: bool

        """

        return all(cage.consistent for cage in self.cages if cell in cage.cells)

For both the Cell and the Cage classes, we have an accept method. This is used to make the objects amenable to the visitor design pattern, for use during solving. The idea is that each cell has a set of “candidate values” that needs to be reduced after we decide to place a value for the cell. I decided to expose things this way to make edits to the core puzzle logic less frequent. Moreover, to try different solution strategies, we need only change the implementation of the visitor we pass to the cells/cages; the core puzzle components need not be changed.

Let’s look at the solver classes:

  • CellVisitor is used to visit cells
  • CageVisitor is used to visit cages; its lifetime is managed by a CellVisitor

And the code:

from utils import with_timing, kk_div, kk_sub


class CellVisitor:
    def __init__(self, candidates, cages):
        """
        Visitor for puzzle cells

        Pass an instance of this object to a puzzle cell
        to "visit" the cell and all the cages that involve
        this cell

        Here we use this object to model the process of eliminating
        a set of candidate values for the given cell

        See https://en.wikipedia.org/wiki/Visitor_pattern
        for more information on this design pattern

        Args:
            candidates: list of cell candidates
            cages: list of cages this visitor should also visit

        """

        self.candidates = candidates
        self.cages = cages

    def __str__(self):
        return '<CellVisitor candidates={0}>'.format(self.candidates)

    def visit_cell(self, cell):
        """
        Visits a `Cell`

        Visit each cage that contains this cell; the resulting
        candidates will be the possible values for this cell

        Args:
            cell: `Cell` object to visit

        Returns: None

        """
        visitor = CageVisitor(self.candidates)
        for cage in self.cages:
            cage.accept(visitor)


class CageVisitor:
    def __init__(self, candidates):
        """
        Visitor for puzzle cages

        The methods in this object are used to prune the cell
        candidate values

        Args:
            candidates: cell candidate values to prune

        """

        self.candidates = candidates

    def __str__(self):
        return '<CageVisitor candidates={0}>'.format(self.candidates)

    def visit_add(self, cage):
        """
        Visits an `AddCage`

        We start with the current cage sum. Any
        value that exceeds the cage target value is pruned

        Args:
            cage: `AddCage` object to visit

        Returns: None

        """
        s = sum(value for value in cage.values if value)
        for value in self.candidates[:]:
            if value + s > cage.value:
                self.prune(value)

    def visit_mul(self, cage):
        """
        Visits a `MulCage`

        Any candidate value that is not a divisor of
        the cage target value is pruned

        Args:
            cage: `MulCage` object to visit

        Returns: None

        """
        for value in self.candidates[:]:
            if cage.value % value != 0:
                self.prune(value)

    def visit_sub(self, cage):
        """
        Visits a `SubCage`

        This implementation removes pairs from the
        candidates if the difference of a given pair
        is not equal to the cage value

        Args:
            cage: `MulCage` object to visit

        Returns: None

        """
        candidates = self.candidates[:]
        for x in candidates:
            if not any(kk_sub([x, y], cage.value) for y in candidates):
                self.prune(x)

    def visit_div(self, cage):
        """
        Visits a `DivCage`

        This implementation removes pairs from the
        candidates if the quotient of a given pair
        is not equal to the cage value

        Args:
            cage: `DivCage` object to visit

        Returns: None

        """
        candidates = self.candidates[:]
        for x in candidates:
            if not any(kk_div([x, y], cage.value) for y in candidates):
                self.prune(x)

    def visit_con(self, cage):
        """
        Visits a `ConCage`

        This implementation removes all candidates
        that are not equal to the cage target value

        Args:
            cage: `ConCage` object to visit

        Returns: None

        """
        for x in self.candidates[:]:
            if x != cage.value:
                self.prune(x)

    def visit_row(self, cage):
        """
        Visits a `RowCage`

        This implementation removes all values
        that are already assigned to a cell in the row

        Args:
            cage: `ConCage` object to visit

        Returns: None

        """
        for value in cage.values:
            self.prune(value)

    def prune(self, value):
        """
        Helper method to safely remove values
        from the candidates

        Args:
            value: to remove

        Returns: None

        """
        if value in self.candidates:
            self.candidates.remove(value)


@with_timing
def backtrack_solve(puzzle):
    """
    Solves a kenken puzzle recursively

    During each iteration of the algorithm, a filtering
    strategy is applied to the puzzle's remaining unassigned cells

    See https://en.wikipedia.org/wiki/Backtracking
    for more information on this algorithm

    Args:
        puzzle: `Puzzle` object to solve

    Returns: bool True if all values in `puzzle` have been assigned a value

    """

    def reduce(cell):
        """
        Reduce the candidate values for this cell

        Args:
            cell: `Cell` object to reduce

        Returns: list of reduced candidates

        """

        candidates = list(puzzle.domain)
        cages = (cage for cage in puzzle.cages if cell in cage.cells)
        cell.accept(CellVisitor(candidates, cages))
        return candidates

    def solve():
        """
        Solve this puzzle recursively

        The algorithm first reduces the candidates for the puzzle's
        unassigned cells

        We then sort the reduced cells by candidate length and
        recursively try values for the current cell until the search
        successfully solves the puzzle

        Returns: bool

        """

        reduced = {cell: reduce(cell) for cell in puzzle.unassigned}

        for cell in sorted(reduced, key=lambda c: len(reduced[c])):
            for value in reduced[cell]:
                cell.value = value

                if puzzle.consistent(cell):
                    if solve():
                        return True

                cell.value = None

            return False
        return puzzle.solved
    return solve()

You can read more about the algorithm in the documentation for the solver. The basic idea is that when we visit a cell, we start off with the puzzle’s full domain. Each of the cages reduces the candidates further, by means of a filtering strategy that is invoked on the candidates when we visit that cage. We do this “reduce” operation for each of the unassigned cells.

Finally, I have a “utils.py” that contains definitions that are in use by the solver and puzzle files. Included is a parse_string method that can be used to create a Puzzle object from a dictionary string:

import time

from ast import literal_eval
from functools import wraps, reduce


def kk_add(values, value):
    """
    Returns whether or not the given values
    sum to the target value

    Args:
        values: list of test values
        value: target value

    Returns: bool

    """
    return sum(values) == value


def kk_mul(values, value):
    """
    Returns whether or not the given values
    multiply to the target value

    Args:
        values: list of test values
        value: target value

    Returns: bool

    """
    return product(values) == value


def kk_sub(values, value):
    """
    Returns whether or not the given values subtract
    to the target value

    Args:
        values: list of test values
        value: target value

    Returns: bool

    """
    return abs(values[0] - values[1]) == value


def kk_div(values, value):
    """
    Returns whether or not the given values divide
    to the target value

    Args:
        values: list of test values
        value: target value

    Returns: bool

    """
    return (int(values[0] / values[1]) == value or
            int(values[1] / values[0]) == value)


def product(nums):
    """
    Helper method to compute the product of a list
    of numbers

    Args:
        nums: list of numbers

    Returns: number

    """
    return reduce(lambda x, y: x * y, nums, 1)


def with_timing(f, output=print):
    """
    Helper method to run a function and output
    the function run time

    Args:
        f: function to decorate
        output: function to output the time message

    Returns: callable decorated function

    """
    @wraps(f)
    def timed(*args, **kwargs):
        ts = time.time()
        result = f(*args, **kwargs)
        te = time.time()

        message = 'func:{!r} args:[{!r}, {!r}] took: {:2.4f} sec'.format(
            f.__name__, args, kwargs, te - ts
        )

        output(message)

        return result
    return timed


def parse_string(s):
    """
    Parse a string to a `Puzzle` object

    The string should be a dictionary that python
    can interpret literally. For example:

    {
      'size': 2,
       'cages': [
         {'value': 2, 'op': '/', 'cells': [(0,0), (0,1)]},
         {'value': 3, 'op': '+', 'cells': [(1,0), (1,1)]}
      ]
    }

    The 'op' should be one of :

        '+' -> AddCage,
        '-' -> SubCage,
        '*' -> MulCage,
        '/' -> DivCage,
        '$' -> ConCage

    The exclusive row/column cages will be created automatically

    Args:
        s: input string to read

    Returns: `Puzzle` object

    """

    from puzzle import (
        Cell,
        AddCage,
        SubCage,
        MulCage,
        DivCage,
        ConCage,
        RowCage,
        Puzzle
    )

    d = literal_eval(s.strip())

    cage_factory = {
        '+': AddCage,
        '-': SubCage,
        '*': MulCage,
        '/': DivCage,
        '$': ConCage
    }

    size = d.get('size')
    cages = d.get('cages')

    if size is None or cages is None:
        raise SyntaxError(
            "Expected 'size' and 'cages'. Got `{0}`".format(d)
        )

    puzzle_cages = []
    puzzle_cells = set()

    for cage in cages:
        value = cage.get('value')
        cells = cage.get('cells')

        if any(cell in puzzle_cells for cell in cells):
            raise ValueError('Some cells exist in another cage {0}'.format(cells))

        if not value or not cells:
            raise SyntaxError(
                "Expected 'value' and 'cells'. Got {0}".format(cage)
            )

        op = cage.get('op')

        if op not in cage_factory:
            raise SyntaxError(
                "Expected '+', '-', '*', '/', '$'. Got {0}".format(op)
            )

        if op == '$' and len(cells) > 1:
            raise ValueError("Expected one cell for `ConstantConstraint`")

        cage_cells = []
        for (row, col) in cells:
            cell = Cell(row, col, None)
            cage_cells.append(cell)

        puzzle_cells = puzzle_cells.union(cage_cells)

        # the constructor for `ConCage` expects a single cell as oppose to a list
        cage = cage_factory[op](cage_cells[0] if op == '$' else cage_cells, value)
        puzzle_cages.append(cage)

    if len(puzzle_cells) != size * size:
        raise Exception(
            'Expected {0} cells; parsed {1}'.format(
                size*size, puzzle_cells)
        )

    for row in range(size):
        cells = [cell for cell in puzzle_cells if cell.row == row]
        puzzle_cages.append(RowCage(cells))

    for col in range(size):
        cells = [cell for cell in puzzle_cells if cell.col == col]
        puzzle_cages.append(RowCage(cells))

    return Puzzle(size, puzzle_cells, puzzle_cages)

Any feedback is welcome. I have some additional puzzle files that I used while debugging/testing the solving algorithm, as well as a “run.py” file which provides a CLI for this application. If you think this is needed, feel free to leave a comment and I can provide a link.


Get this bounty!!!