#StackBounty: #python #django #orm Multiple left join and left join against the same model raw query convert to Django ORM

Bounty: 50

Using Django 1.11

I have the following models:

class Vendor(Model):

class Product(Model):

class Pricebook(Model):

class Quote(Model):
     vendor = models.ForeignKey(Vendor)

class SKU(Model):
     product = models.ForeignKey(Product)
     pricebook = models.ForeignKey(Pricebook)
     vendor    = models.ForeignKey(Vendor)

class SKUPrice(Model):
    sku = models.ForeignKey(SKU, related_name="prices")

class LineItem(Model):
    quote = models.ForeignKey(Quote, related_name="quote_line_items")
    sku = models.ForeignKey(SKU)

This is the raw query that works for me.

SELECT 
  qli.quantity, 
  sku_source.product_id, 
  sku_dest.id as sku_dest_id, 
  sku_dest_price.id as sku_dest_price_id 
FROM lineitem qli 
INNER JOIN sku sku_source ON 
  qli.sku_id = sku_source.id 
LEFT JOIN sku sku_dest ON 
  sku_dest.pricebook_id = sku_source.pricebook_id AND
  sku_dest.product_id = sku_source.product_id 
LEFT JOIN skuprice sku_dest_price ON
  sku_dest_price.status = 'default' AND 
  sku_dest_price.sku_id = sku_dest.id
WHERE qli.quotation_id = 40 AND 
  qli.quantity > 0 AND 
  sku_dest.vendor_id = 38;

What I have tried is:

(the_quote_with_id_as_40
.quotation_line_items
.filter(quantity__gt=0)
.select_related('sku__product')
.values('sku__product__id', 'quantity')
)

This produces this query

SELECT "sku"."product_id", "lineitem"."quantity"
FROM "lineitem" 
INNER JOIN "sku" ON ("lineitem"."sku_id" = "sku"."id") 
WHERE ("lineitem"."quotation_id" = 40 AND 
       "lineitem"."quantity" > 0)

which is not exactly what I want.

I can of course use raw query. But I would like to know if possible to use ORM. At the very least, this is for expanding my knowledge.

Thank you.

UPDATE

Since there’s a request to clarify my models, I am writing the following.

I have a Quote object. It allows many LineItem objects. Each LineItem is one-to-many with SKU and Quote.

A SKU belongs to a Pricebook and a Product and a Vendor. These description of the relation can also be gleaned from the above code.

But for clarity, I will repeat here.

This arrangement is such that a single, same Product can be sold by different Vendors but they will appear as different SKUs.

This is by design.

Our use case is this: a user attempts to duplicate the LineItems of a particular Quote to a different Quote.

In the mind of the user, they don’t have the sophistication to appreciate the differences between a SKU, a LineItem, or a Product.

It’s all Product in their mind. They just want the same Product to appear in a different Quote bearing the same quantity.

The challenge is this. We now have two Quotes (a source Quote to copy from and a destination Quote to copy to). Both Quotes may have the same Vendor. Or they may not. The user wants my Django app to automatically cater for both situations.

So this means I need to find out the following before I duplicate.

  • the quantity as stated in the LineItem
  • the Product as stated in the LineItem vis-a-vis the SKU
  • find the corresponding SKU sold by the Vendor for the destination Quote
  • and its corresponding unit price

The raw query allows me to extract all 4 pieces of information in a single query. It’s efficient.

Hence you see that I have the alias sku_source and sku_dest

This is why my WHERE criteria contains 3 pieces of conditions:

  1. only Source LineItems with quantity > 0 will be copied
  2. only LineItems from a particular source Quote will be copied
  3. only SKUs meant for a particular Vendor (in this case the Vendor for the destination Quote) will be looked into

It’s possible that:

  1. multiple LineItems for the same Product (vis-a-vis SKU) will appear in the same source Quote.

  2. the destination Vendor (meaning the Vendor of the destination Quote) does not sell a particular Product that the source Vendor does sell. Hence I use LEFT JOIN. This means this particular Product will not be duplicated.

I hope this clears things up.


Get this bounty!!!

#StackBounty: #django #celery #celerybeat How to solve celerybeat is down: no pid file found?

Bounty: 50

I have followed instructions from https://pythad.github.io/articles/2016-12/how-to-run-celery-as-a-daemon-in-production
It works pretty well for celeryd, however when starting celerybeat it says pid file not found.

I’ve used this tutorial on my previous projects and it did fine for both celeryd and celerybeat. The difference of this project only is all project files including the django project are owned by root. I fail at finding more details about the issue.


Get this bounty!!!

#StackBounty: #django #django-admin Django admin failing to login after upgrade to 2.1

Bounty: 50

After upgrading an old Django 1.8 to 2.1, when I try to login on my admin site, I get a 404 with message:

Using the URLconf defined in <mysite>.urls, Django tried these URL patterns, in this order:
[...]
The current path, login/, didn't match any of these.

Which I guess is true because it should be on __admin/login as in my urls.py I have:

urlpatterns = [
    ...
    path(r'__admin/', admin.site.urls),
    ...
]

But:

  • The GET request /__admin/login returns the login page as expected
  • The problem happens only in production (with passenger WSGI on a VPS), not on localhost
  • Unfortunately the upgrade happened at the same time as a migration in VPS machine so there may also be a problem with the database.


Get this bounty!!!

#StackBounty: #django #django-models #django-rest-framework Reduce Django serialization time

Bounty: 50

I am doing queries for roughly 100,000 rows with approximately 40 column each. The columns are of combination of float, integer, datetime, and char.

The query time is about two seconds, and serialization is taking forty seconds or more, whereas response building is about two seconds too.

I am wondering how can I reduce serialization time for Django models?

Here is my model:

class TelematicsData(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)

    device = models.ForeignKey(Device, on_delete=models.CASCADE, null=True)

    created_date = models.DateTimeField(auto_now=True)

    analog_input_01 = models.FloatField(null=True)
    analog_input_02 = models.FloatField(null=True)
    analog_input_03 = models.FloatField(null=True)
    analog_input_04 = models.FloatField(null=True)
    analog_input_05 = models.FloatField(null=True)
    analog_input_06 = models.FloatField(null=True)

    device_temperature = models.FloatField(null=True)
    device_voltage = models.FloatField(null=True)
    vehicle_voltage = models.FloatField(null=True)

    absolute_acceleration = models.FloatField(null=True)
    brake_acceleration = models.FloatField(null=True)
    bump_acceleration = models.FloatField(null=True)
    turn_acceleration = models.FloatField(null=True)
    x_acceleration = models.FloatField(null=True)
    y_acceleration = models.FloatField(null=True)
    z_acceleration = models.FloatField(null=True)

    cell_location_error_meters = models.FloatField(null=True)
    engine_ignition_status = models.NullBooleanField()

    gnss_antenna_status = models.NullBooleanField()
    gnss_type = models.CharField(max_length=20, default='NA')
    gsm_signal_level = models.FloatField(null=True)
    gsm_sim_status = models.NullBooleanField()

    imei = models.CharField(max_length=20, default='NA')
    movement_status = models.NullBooleanField()
    peer = models.CharField(max_length=20, default='NA')

    position_altitude = models.IntegerField(null=True)
    position_direction = models.FloatField(null=True)
    position_hdop = models.IntegerField(null=True)
    position_latitude = models.DecimalField(max_digits=9, decimal_places=6, null=True)
    position_longitude = models.DecimalField(max_digits=9, decimal_places=6, null=True)
    position_point = models.PointField(null=True)
    position_satellites = models.IntegerField(null=True)
    position_speed = models.FloatField(null=True)
    position_valid = models.NullBooleanField()

    shock_event = models.NullBooleanField()
    hardware_version = models.FloatField(null=True)
    software_version = models.FloatField(null=True)

    record_sequence_number = models.IntegerField(null=True)
    timestamp_server = models.IntegerField(null=True)
    timestamp_unix = models.IntegerField(null=True)
    timestamp = models.DateTimeField(null=True)
    vehicle_mileage = models.FloatField(null=True)

    user_data_value_01 = models.FloatField(null=True)
    user_data_value_02 = models.FloatField(null=True)
    user_data_value_03 = models.FloatField(null=True)
    user_data_value_04 = models.FloatField(null=True)
    user_data_value_05 = models.FloatField(null=True)
    user_data_value_06 = models.FloatField(null=True)  
    user_data_value_07 = models.FloatField(null=True)  
    user_data_value_08 = models.FloatField(null=True)

and this is the serializer:

class TelematicsDataSerializer(serializers.ModelSerializer):

    class Meta:
        model = TelematicsData
        geo_field = ('position_point')
        #fields = '__all__'
        exclude = ['id']


Get this bounty!!!

#StackBounty: #python #django #oop Create a Blog which support multiple type of post

Bounty: 50

I am a new user of Django, and I am trying to figure out how to created a model which can support many kind (type) of elements.

This is the plot : I want to create a Blog module on my application.
To do this, I created a model Page, which describe a Blog Page. And a model PageElement, which describe a Post on the blog. Each Page can contain many PageElement.

A PageElement can have many types, because I want my users could post like just a short text, or just a video, or just a picture. I also would like (for example) the user could just post a reference to another model (like a reference to an user). Depending of the kind of content the user posted, the HTML page will display each PageElement in a different way.

But I don’t know what is the right way to declare the PageElement class in order to support all these cases 🙁

Here is my Page model :

class Page(models.Model):
    uuid = models.UUIDField(default=uuid.uuid4, editable=False, unique=True)

    # Basical informations
    title = models.CharField(max_length=150)
    description = models.TextField(blank=True)

    # Foreign links
    user = models.ForeignKey(
        settings.AUTH_USER_MODEL,
        on_delete=models.SET_NULL,
        null=True,
        related_name='pages_as_user'
    )

    created_at = models.DateTimeField(default=timezone.now)

    # Other fields ....

    class Meta:
        indexes = [
            models.Index(fields=['uuid']),
            models.Index(fields=['user', 'artist'])
        ]

For now, I have two solutions, the first one use inheritance : When you create a new post on the blog, you create an Element which inherit from PageElement model. Here are my different Models for each cases :

class PageElement(models.Model):
    page = models.OneToOneField(
        Page,
        on_delete=models.CASCADE,
        related_name='%(class)s_elements'
    )

    updated_at = models.DateTimeField(default=timezone.now)
    created_at = models.DateTimeField(default=timezone.now)

class PageImageElement(PageElement):
    image = models.ImageField(null=True)
    image_url = models.URLField(null=True)

class PageVideoElement(PageElement):
    video = models.FileField(null=True)
    video_url = models.URLField(null=True)

class PageTextElement(PageElement):
    text = models.TextField(null=True)

class PageUserElement(PageElement):
    user = models.ForeignKey(
        'auth.User',
        on_delete=models.CASCADE,
        related_name='elements'
    )

This solution would be the one I have choosen if I had to work if a “pure” Django. Because I could stored each PageElement in a dictionnary and filter them by class. And this solution could be easily extended in the futur with new type of content.

But with Django models. It seems that is not the best solution. Because it will be really difficult to get all PageElement children from the database (I can’t just write “page.elements” to get all elements of all types, I need to get all %(class)s_elements elements manually and concatenate them :/). I have thinked about a solution like below (I don’t have tried it yet), but it seems overkilled for this problem (and for the database which will have to deal with a large number of request):

class Page(models.Model):
    # ...
    def get_elements(self):
        # Retrieve all PageElements children linked to the current Page
        R = []
        fields = self._meta.get_fields(include_hidden=True)
        for f in fields:
            try:
                if 'conversation_elements' in f.name:
                    R += getattr(self, f.name)
            except TypeError as e:
                continue

        return R

My second “solution” use an unique class which contains all fields I need. Depending of the kind of PageElement I want to create, I would put type field to the correct value, put the values in the corresponding fields, and put to NULL all other unused fields :

class PageElement(models.Model):
    page = models.OneToOneField(
        Page,
        on_delete=models.CASCADE,
        related_name='elements'
    )

    updated_at = models.DateTimeField(default=timezone.now)
    created_at = models.DateTimeField(default=timezone.now)

    TYPES_CHOICE = (
        ('img', 'Image'),
        ('vid', 'Video'),
        ('txt', 'Text'),
        ('usr', 'User'),
    )
    type = models.CharField(max_length=60, choices=TYPES_CHOICE)

    # For type Image
    image = models.ImageField(null=True)
    image_url = models.URLField(null=True)

    # For type Video
    video = models.FileField(null=True)
    video_url = models.URLField(null=True)

    # For type Text
    text = models.TextField(null=True)

    # For type User
    user = models.ForeignKey(
        'auth.User',
        on_delete=models.CASCADE,
        related_name='elements',
        null=True
    )

With this solution, I can retrieve all elements in a single request with “page.elements”. But it is less extendable than the previous one (I need to modify my entire table structure to add a new field or a new kind of Element).

To be honnest, I have absolutly no idea of which solution is the best. And I am sure other (better) solutions exist, but my poor Oriented-Object skills don’t give me the ability to think about them ( 🙁 )…

I want a solution which can be easily modified in the future (if for example, I want to add a new Type “calendar” on the Blog, which reference a DateTime). And which would be easy to use in my application if I want to retrieve all Elements related to a Page…

Thanks for your attention 🙂


Get this bounty!!!

#StackBounty: #python #django #session #cookies #redis Django: Sessions not working, users continually being logged out and prompted to…

Bounty: 100

I want to keep users on a Heroku app logged in, and they are currently being logged out on every view that requires login. The problem seems to be related to the sessionid in the Response Cookie not being set. E.g., clicking on “Account” will redirect users to the login view.

Response Headers

Location: /login?next=/home/
Set-Cookie: sessionid=; Domain=.appname.com; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/

settings.py

SESSION_ENGINE = 'django.contrib.sessions.backends.cached_db'
SESSION_SAVE_EVERY_REQUEST = True
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
SESSION_COOKIE_DOMAIN = ".appname.com"
SESSION_COOKIE_AGE = 2419200
SESSION_COOKIE_HTTPONLY = False
SESSION_COOKIE_SECURE = True

CSRF_COOKIE_SECURE = True
CSRF_COOKIE_DOMAIN = ".appname.com"

SECURE_SSL_REDIRECT = True

# ...

MIDDLEWARE_CLASSES = [
    'django.middleware.cache.UpdateCacheMiddleware',

     # ...

    'django.middleware.cache.FetchFromCacheMiddleware',
]

# ...

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'appname',
    }
}

# https://devcenter.heroku.com/articles/python-concurrency-and-database-connections
db_from_env = dj_database_url.config(conn_max_age=500)
DATABASES['default'].update(db_from_env)

# https://devcenter.heroku.com/articles/heroku-redis
CACHES = {
    "default": {
         "BACKEND": "redis_cache.RedisCache",
         "LOCATION": os.environ.get('REDIS_URL'),
    }
}

views.py

def login(request):
    if hasattr(request, 'session') and not request.session.session_key:
        request.session.save()
        request.session.modified = True
    if request.user.is_authenticated():
        return redirect('home')
    else:
        return render(request, "login.html", {})

@login_required
def home(request):
    return render(request, "home.html", {})

@require_POST
def ajax_login(request):
    data = {}

    if request.method == 'POST':
        form = LoginForm(request.POST)

        if form.is_valid():
            email = form.cleaned_data['email']
            password = form.cleaned_data['password']

        user = authenticate(username=email, password=password)
        if user is not None:
            login(request, user)
            data = {
                'is_authenticated': True
            }
    else:
        form = LoginForm()
        data = {
            'is_authenticated': False
        }
    return JsonResponse(data)

Here is how the app makes AJAX calls.

app.js

$.ajaxSetup({
  beforeSend: function(xhr, settings) {
    if (!csrfSafeMethod(settings.type) && !this.crossDomain) {
      xhr.setRequestHeader("X-CSRFToken", csrftoken);
    }
  }
});
$.ajax({
  url: '/ajax/login',
  type: 'POST',
  data: {
    'username': username,
    'password': password,
  },
  dataType: 'json',
  credentials: 'include',
  success: function (data) {
    if (data.is_authenticated) {
      window.location.href = "/home/";
    }
  }
});

app.html


var csrftoken = $("[name=csrfmiddlewaretoken]").val();


Get this bounty!!!

#StackBounty: #python #django #django-rest-framework Django Rest Framework – passing Model data through a function, then posting output…

Bounty: 50

(Django 2.0, Python 3.6, Django Rest Framework 3.8)

I’m trying to fill the calendarydays field in the model below:

Model

class Bookings(models.Model):
    booked_trainer = models.ForeignKey(TrainerProfile, on_delete=models.CASCADE)
    booked_client = models.ForeignKey(ClientProfile, on_delete=models.CASCADE)
    trainer_availability_only = models.ForeignKey(Availability, on_delete=models.CASCADE)
    calendarydays = models.CharField(max_length=300, blank=True, null=True)

    PENDING = 'PENDING'
    CONFIRMED = 'CONFIRMED'
    CANCELED = 'CANCELED'

    STATUS_CHOICES = (
        (PENDING, 'Pending'),
        (CONFIRMED, 'Confirmed'),
        (CANCELED, 'Canceled')
    )


    booked_status = models.CharField(
        max_length = 9,
        choices = STATUS_CHOICES,
        default = 'Pending'
    )

    def __str__(self):
        return str(self.trainer_availability_only)

Now, I have a function that takes values from trainer_availability_only and converts those values to a list of datetime strings, the returned output would look like this:

{‘calendarydays’: [‘2018-07-23 01:00:00’, ‘2018-07-23 02:00:00’,
‘2018-07-23 03:00:00’, ‘2018-07-30 01:00:00’, ‘2018-07-30 02:00:00’,
‘2018-07-30 03:00:00’, ‘2018-08-06 01:00:00’, ‘2018-08-06 02:00:00’,
‘2018-08-06 03:00:00’, ‘2018-08-13 01:00:00’, ‘2018-08-13 02:00:00’,
‘2018-08-13 03:00:00’, ‘2018-08-20 01:00:00’, ‘2018-08-20 02:00:00’,
‘2018-08-20 03:00:00’]}

Problem

How can I fill the calendarydays field with the function output for a user to select from a dropdown, and where should I implement this logic (in my view or the serializer)? My main point of confusion is that, because my function depends on data from trainer_availability_only, I don’t want to create a separate model/table for this information (as that would seem too repetitive). I also don’t fully understand where in my serializers or views I can implement some sort of dropdown for a User to choose a single calendarydays value for (like I would be able to for a ForeignKey or OneToOneField for example).

Details for the other models aren’t really relevant to the question, except trainer_availability_only, which basically gives the user a dropdown selection that would look like this:

('Monday','12:00 am - 1:00 am')
('Wednesday','4:00 pm - 5:00 pm')
etc.

Any help is greatly appreciated.


Get this bounty!!!

#StackBounty: #python #django #amazon-web-services #deployment #elastic-beanstalk AWS Elastic Beanstalk health check issue

Bounty: 50

First I apologize for not being familiar with English.

My web application is Django and web server use Nginx, use Docker image and Elastic Beanstalk to deployment.

Normally there was no problem, but as the load balancer expands EC2, my web server becomes 502 Bad Gateway.

I’m checked Elastic Beanstalk application logs, about 16% of the requests returned 5xx errors, at which time the load balancer expands EC2, causing the web server to transition to the 502 Bad Gateway state and the Elastic Beanstalk application to the Degraded state.

Is this a common problem when the load balancer performs a health check? If not, can I tell you how to turn off the Health Check?

I will attach a capture image for your help.

enter image description here


Get this bounty!!!

#StackBounty: #python #django #django-models #django-1.8 #django-1.9 Using a model's sub-classes as choice options for that model r…

Bounty: 100

We are trying to upgrade a legacy code’s django version from 1.8 to 1.9. We have one model that is defined like this:

def _get_descendant_question_classes():
    stack = [Question]

    while stack:
        cls = stack.pop()
        stack.extend(cls.__subclasses__())
        yield cls

def _get_question_choices():
    question_classes = _get_descendant_question_classes()

    for cls in question_classes:
        yield (cls.slug, cls._meta.verbose_name)

class Question(models.Model):
    type = models.CharField(max_length=10, choices=_get_question_choices(), default=slug)

class TextQuestion(Question): pass
class SelectQuestion(Question): pass
...

Basically the model wants to use its sub-classes as choice options for one of its fields. It does this with traversing the model in a DFS manner and yielding all the sub-classes.

This code works in django 1.8 but in django 1.9 it gives this error:

Traceback (most recent call last):
  File "./manage.py", line 16, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 350, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 324, in execute
    django.setup()
  File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
    app_config.import_models(all_models)
  File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/saeed/saeed/survey/models.py", line 85, in <module>
    class Question(models.Model):
  File "/home/saeed/saeed/survey/models.py", line 99, in Question
    type = models.CharField(max_length=10, choices=_get_question_choices(), default=slug)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 1072, in __init__
    super(CharField, self).__init__(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 161, in __init__
    choices = list(choices)
  File "/home/saeed/saeed/survey/models.py", line 65, in _get_question_choices
    for cls in question_classes:
  File "/home/saeed/saeed/survey/models.py", line 54, in _get_descendant_question_classes
    stack = [Question]
NameError: global name 'Question' is not defined

I understand the problem what I don’t understand is how this works in django 1.8? What has changed in django 1.9 that causes this? What is the best way to fix this?


Get this bounty!!!

#StackBounty: #python #django #django-i18n "django-admin.py makemessages -l en" adds Plural-Forms to the output file

Bounty: 50

Every time I run django-admin.py makemessages -l en, it adds the following line to the djangojs.po file:

"Plural-Forms: nplurals=INTEGER; plural=EXPRESSION;n"

after that running python manage.py runserver break out whith this error:

ValueError: plural forms expression could be dangerous

Of course removing that line fix the error and make it go away.
How can I prevent this line from being added?


Get this bounty!!!