#StackBounty: Efficient extraction of patch features over an image

Bounty: 50

Input:

  • data – a matrix of size n*m*3*3 (complex values)
  • indices – a list of coordinates (x,y), where x < n and y < m
  • fp – a feature parameter which is a tuple of ((fp11, fp12), (fp21, fp22)), id)
  • reference – a list of 3*3 matrices
  • swrd – a function which computes a similarity value between two complex valued 3*3 matrices

Output:

  • feature_values – a list of features – one feature for each index in (indices)

Functionality:

Given an image (data) were each pixel is a 3*3 matrix. And there is a list of target pixels (indices). For each target pixel, I want to extract features of the patch surrounding it.

A patch feature is either:
a) the swrd of a pixel in the patch with a reference matrix or
b) the swrd of two pixels in the patch

Thus a feature can be described by the relative coordinates fp11, fp12 (x and y offset of pixel of interest 1) and fp21, fp22 (x and y offset of pixel of interest 2).
If fp11 == fp21 and fp12== fp22, then i want to compute a), else i want to compute b).
The reference matrix of interest is defined by the feature parameter called id.

Note that the indices of interest are already filtered so that the sum x+fp__ < n and y+fp__< m for all possible fp__.

Code

Computing the symetric revised wishart distance with regularization in case a matrix A or B is not invertible

def srwd(A, B):
    """This function computes the symetric revised wishart distance as from the paper
    SPECTRAL CLUSTERING OF POLARIMETRIC SAR DATA WITH WISHART-DERIVED DISTANCE MEASURES"""
    try:
        dist = 0.5 * np.trace(np.dot(A, inv(B)) + np.dot(B, inv(A))) - len(A)      
    except:
        A, B = A.reshape(3, 3) + np.eye(3) * 0.000001, B.reshape(3, 3) + np.eye(3) * 0.000001
        dist = 0.5 * np.trace(np.dot(A, inv(B)) + np.dot(B, inv(A))) - len(A)      
    return abs(dist)

Getting the features with the input as given above:

def feature(data, indices, fp, reference):
    # fp is a tuple of 2 coordinates in a patch ((x1,x2),(y1,y2),ref),
    # where ref is an index of a random reference matrix in reference only relevant in case x1=y1 and x2=y2
    res = []
    if fp[0] != fp[1]:
        for i in indices:
            x, y = i
            res.append(srwd(data[x + fp[0][0]][y + fp[0][1]], data[x + fp[1][0]][y + fp[1][1]]))
    else:
        for i in indices:
            x, y = i
            res.append(srwd(data[x + fp[0][0]][y + fp[0][1]], reference[fp[2]]))
    return res

Finally there is another loop such as:

for fp in feature_params:
    feature_values = feature(data, indices, fp, reference)
    #here work on feature_values

The current implementation is rather inefficent and a bottleneck of the whole process. How could I improve it?

Is there a chance to efficiently compute a feature matrix efficiently and operate on it afterwards?

An executable toy example including the whole code is given here (allowing copy-paste)

import numpy as np
from numpy.linalg import inv

#toy example
data = np.random.rand(1000, 1000, 3, 3) #an image of 1000*1000 pixels, each pixel a 3*3 matrix
indices = np.random.randint(3,96, size = (10000,2)) # a list of 10000 target pixels (lets assume they are unique)
reference = [np.random.rand(3,3)] # a single reference matrix in a list (in actual application there are multiple reference matrices)
feature_params = [((0,0),(-1,-1), 0), ((0,0), (0,0), 0), ((0,1), (0,0), 0), ((1,0), (0,0), 0), ((1,1), (0,0), 0)] 



def srwd(A, B):
    """This function computes the symetric revised wishart distance as from the paper
    SPECTRAL CLUSTERING OF POLARIMETRIC SAR DATA WITH WISHART-DERIVED DISTANCE MEASURES"""
    try:
        dist = 0.5 * np.trace(np.dot(A, inv(B)) + np.dot(B, inv(A))) - len(A)      
    except:
        A, B = A.reshape(3, 3) + np.eye(3) * 0.000001, B.reshape(3, 3) + np.eye(3) * 0.000001
        dist = 0.5 * np.trace(np.dot(A, inv(B)) + np.dot(B, inv(A))) - len(A)      
    return abs(dist)


def feature(data, indices, fp, reference):
    # fp is a tuple of 2 coordinates in a patch ((x1,x2),(y1,y2),ref),
    # where ref is an index of a random reference matrix in reference only relevant in case x1=y1 and x2=y2
    res = []
    if fp[0] != fp[1]:
        for i in indices:
            x, y = i
            res.append(srwd(data[x + fp[0][0]][y + fp[0][1]], data[x + fp[1][0]][y + fp[1][1]]))
    else:
        for i in indices:
            x, y = i
            res.append(srwd(data[x + fp[0][0]][y + fp[0][1]], reference[fp[2]]))
    return res

for fp in feature_params:
    feature_values = feature(data, indices, fp, reference)        
    #here work on feature_values

A final note about the dimensions of the actual problem:

  • Image of size 6000*1700,
  • around 500 features in feature_params
  • indices is list of around 8.000.000 target indices


Get this bounty!!!

Launch HTML code in browser from Python

Lets say you have generated some html content for a web page dynamically and have it in memory variable in python.

In order to view and test this content, you would need a Python program that prints out the HTML code, which then would have to be copied and pasted to a HTML file, then from there, you can test it in a browser.

In Python, there is a way to launch such HTML code in a web browser so that you don’t have to go through the copy and paste method every time

Using webbrowser.open:

Source

Sort a list of tuples by Nth item in Python

Suppose you have a list of tuples that looks something like this:

[('abc', 121),('abc', 231),('abc', 148), ('abc',221)]

And you want to sort this list in ascending order by the integer value inside the tuples.

We can achieve this using the key keyword with sorted().

sorted([('abc', 121),('abc', 231),('abc', 148), ('abc',221)], key=lambda x: x[1])

key should be a function that identifies how to retrieve the comparable element from your data structure. For example, the second element of the tuple, so we access [1].

Source: StackOverflow.com

The Fuck

Magnificent app which corrects your previous console command, inspired by a @liamosaur tweet.

Few more examples:

➜ apt-get install vim
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

➜ fuck
sudo apt-get install vim [enter/↑/↓/ctrl+c]
[sudo] password for nvbn:
Reading package lists... Done
...
➜ git push
fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use

    git push --set-upstream origin master


➜ fuck
git push --set-upstream origin master [enter/↑/↓/ctrl+c]
Counting objects: 9, done.
...
➜ puthon
No command 'puthon' found, did you mean:
 Command 'python' from package 'python-minimal' (main)
 Command 'python' from package 'python3' (main)
zsh: command not found: puthon

➜ fuck
python [enter/↑/↓/ctrl+c]
Python 3.4.2 (default, Oct  8 2014, 13:08:17)
...
➜ git brnch
git: 'brnch' is not a git command. See 'git --help'.

Did you mean this?
    branch

➜ fuck
git branch [enter/↑/↓/ctrl+c]
* master
➜ lein rpl
'rpl' is not a task. See 'lein help'.

Did you mean this?
         repl

➜ fuck
lein repl [enter/↑/↓/ctrl+c]
nREPL server started on port 54848 on host 127.0.0.1 - nrepl://127.0.0.1:54848
REPL-y 0.3.1
...

Source

Hacker Scripts..Based on a true story

An worker who had left the company had some scripts to automate his life..his coworker has posted this gem to a russian bash.org.

Based on a true story:

xxx: OK, so, our build engineer has left for another company. The dude was literally living inside the terminal. You know, that type of a guy who loves Vim, creates diagrams in Dot and writes wiki-posts in Markdown… If something – anything – requires more than 90 seconds of his time, he writes a script to automate that.

xxx: So we’re sitting here, looking through his, uhm, “legacy”

xxx: You’re gonna love this

xxx: smack-my-bitch-up.sh – sends a text message “late at work” to his wife (apparently). Automatically picks reasons from an array of strings, randomly. Runs inside a cron-job. The job fires if there are active SSH-sessions on the server after 9pm with his login.

xxx: kumar-asshole.sh – scans the inbox for emails from “Kumar” (a DBA at our clients). Looks for keywords like “help”, “trouble”, “sorry” etc. If keywords are found – the script SSHes into the clients server and rolls back the staging database to the latest backup. Then sends a reply “no worries mate, be careful next time”.

xxx: hangover.sh – another cron-job that is set to specific dates. Sends automated emails like “not feeling well/gonna work from home” etc. Adds a random “reason” from another predefined array of strings. Fires if there are no interactive sessions on the server at 8:45am.

xxx: (and the oscar goes to) fucking-coffee.sh – this one waits exactly 17 seconds (!), then opens an SSH session to our coffee-machine (we had no frikin idea the coffee machine is on the network, runs linux and has SSHD up and running) and sends some weird gibberish to it. Looks binary. Turns out this thing starts brewing a mid-sized half-caf latte and waits another 24 (!) seconds before pouring it into a cup. The timing is exactly how long it takes to walk to the machine from the dudes desk.

xxx: holy sh*t I’m keeping those

Source

Github scripts

Enable/Disable Proxy Service in OSB via ANT+Python

Below is the configurable code for developers to maintain an ant script to be able to control(Enable/Disable) Proxy Services via Ant Script.

Files:


WLSTbuild.properties

weblogic.home.dir=<OSB Home>/wlserver_10.3
weblogic.lib.dir=${weblogic.home.dir}/server/lib


build.properties

fmw.home=<OSB Middleware Home>
wls.username =<Weblogic User Id>
wls.password =<Weblogic Password>
wls.server = t3://:<Weblogic Host>:<Weblogic Port>
service.script=osb_enable_service.py
service.project="<Project Name>/<Sub Project Name or 'ProxyServices' >"
service.name =<Proxy Name>
service.action = <disable/enable>


build.xml

<?xml version="1.0" encoding="windows-1252" ?>
<project name="ServiceControl" default="ServiceControl">
   <property file="./build.properties"/>
   <property file="./wlstbuild.properties"/>
   <taskdef classname="weblogic.ant.taskdefs.management.WLSTTask" name="wlst">
      <classpath>
         <pathelement location="${weblogic.lib.dir}/weblogic.jar"></pathelement>
      </classpath>
   </taskdef>
   <target name="ServiceControl">
      <echo>${weblogic.lib.dir}</echo>
      <wlst filename="${service.script}" debug="true" failonerror="false"
            arguments="${wls.username} ${wls.password} ${wls.server} ${service.project} ${service.name} ${service.action}">
         <classpath>
            <pathelement location="${weblogic.lib.dir}/weblogic.jar"></pathelement>
            <pathelement location="${fmw.home}/<OSB Installation Folder>/lib/sb-kernel-api.jar"></pathelement>
            <pathelement location="${fmw.home}/<OSB Installation Folder>/lib/sb-kernel-impl.jar"></pathelement>
            <pathelement location="${fmw.home}/<OSB Installation Folder>/lib/osb-coherence-client.jar"></pathelement>
         </classpath>
      </wlst>
   </target>
</project>

osb_enable_service.py

from com.bea.wli.sb.management.configuration import SessionManagementMBean
from com.bea.wli.sb.management.configuration import ALSBConfigurationMBean
from com.bea.wli.config import Ref
from java.lang import String
from com.bea.wli.monitoring import StatisticType
from com.bea.wli.config import Ref
from com.bea.wli.sb.util import Refs
from com.bea.wli.sb.management.configuration import CommonServiceConfigurationMBean
from java.lang import String
from com.bea.wli.monitoring import StatisticType
from com.bea.wli.config import Ref
from com.bea.wli.sb.util import Refs
from com.bea.wli.sb.management.configuration import CommonServiceConfigurationMBean
adminUser = sys.argv[1]
adminPassword = sys.argv[2]
adminUrl = sys.argv[3]
project = sys.argv[4]
service = sys.argv[5]
action = sys.argv[6]
connect(adminUser,adminPassword,adminUrl)
domainRuntime()
print('WLST Connected')

Create a session name

sessionName = String("SessionScript"+Long(System.currentTimeMillis()).toString())
print('... after session. Session is: ', sessionName)

Get the session MBean and create a session

sessionMBean = findService(SessionManagementMBean.NAME,SessionManagementMBean.TYPE)
print('SessionMBean is: ', sessionMBean)
sessionMBean.createSession(sessionName)
print(String('Session was created ... ').concat(sessionName))

Get the ProxyServiceConfigurationMBean specific to our session

mbean = findService(String("ProxyServiceConfiguration.").concat(sessionName),'com.bea.wli.sb.management.configuration.ProxyServiceConfigurationMBean')

Creates a reference to a folder and 'ProxyServices' is a folder in the project and 'ProxyServices' is a folder in the project

projectName = Refs.makeParentRef(project + '/')
proxyRef = Refs.makeProxyRef(projectName, service)
if action == 'disable' :
print "Disabling " + service +" has been completed"
mbean.disableService(proxyRef)
else :
print "Enabling " + service +" has been completed"
mbean.enableService(proxyRef)

Now commit (activate) the changes

sessionMBean.activateSession(sessionName, "disabled PS")
print
disconnect()
exit()

Place all files in a single folder.

From CMD, Navigate to folder and just call ant

Verify from OSB Portal, the service will be in disabled/enabled format in Operational Settings Tab.