dis-moi / backend Goto Github PK

#20

@lilobase

on va utiliser un format intermédiaire
que l’on hydratera depuis un resultset de la DB
ça va nous faire gagner des perfs
et permettre une forme consistante
et plus de flexibilité dans ce que l’on renvoie (ie, les url en absolue par exemple)

Will solve #4
/!\ Assume the extension will request sub-objects collection on-demand from a list of ids

Related to #14

From the back-office Only Super users should be able to view and edits:

• Users
• Contributors
• Organizations

Also, whenever a contributor creates a new recommendation, the Contributor of the Recommendation should be automatically set to himself.

Last but not least, only Contributors-Editors are allowed to change the Contributor of a Recommendation, from a list of Contributors belonging to the same Organization (i.e. the contributor-editor’s organization).

NB : de mémoire cette fonctionnalité avait été commencée par @JalilArfaoui

Actuellement, lorsqu'un utilisateur crée une bulle depuis l'extension, un mail est généré et traité par l'équipe éditoriale via des copiés collés.

La présente évolution vise à ce que la contribution soit directement pré-intégrée dans le BO.
L'alerte mail demeure ne fonctionnalité existante afin de prévenir l'équipe éditoriale d'une contribution à traiter.

On production, logs are written into a bucket file...
On staging, logs are... ignored?

126-disconnection-issue-token-session-or-clevercloud-autoscaler.

Investigation results

• By default, clevercloud supports the session sharing between instances via a "replicated file system" ;
• the app config does not use the default session handling :

# app/config/config.yml
framework:
    # ...
    session:
        # http://symfony.com/doc/current/reference/configuration/framework.html#handler-id
        handler_id:  session.handler.native_file
        save_path:   "%kernel.root_dir%/../var/sessions/%kernel.environment%"

Since I cannot find more detailed data on how clevercloud actually handle the "replicated file system", I suppose it only supports the default session directory.

Possible solutions

• Understand why the app use a non standard session handling and safely remove it ;
• Use "custom" session handling supporting multiple instances, like memcache/redis or db ;

See #80 (review)

Please cleanup unused git branches

Whenever one deletes a recommendation, its criteria get deleted as well.

Contributors can click a logout button to logout from the backoffice. As a result, they won’t see their private recommendations (which is sometimes desirable), until they login again.

There must be something wrong with some notices since the notices’ list pagination fails on some page:

• fine: https://notices.lmem.net/admin/?entity=Notice&action=list&menuIndex=0&submenuIndex=-1&sortField=id&sortDirection=DESC&page=3
• broken: https://notices.lmem.net/admin/?entity=Notice&action=list&menuIndex=0&submenuIndex=-1&sortField=id&sortDirection=DESC&page=2

Don’t know if it’s related to #2 but I’m unable to delete some reco.

For instance, this ends up with an error 500 (reco #15): http://lmem-craft-backend.cleverapps.io/admin/?action=delete&entity=Recommendation&sortField=id&sortDirection=DESC&page=1&referer=%252Fadmin%252F%253Faction%253Dlist%2526entity%253DRecommendation%2526sortField%253Did%2526sortDirection%253DDESC%2526page%253D1&id=15

You can reproduce it on any reco, with or without a source being set.

There is no DTO handling for this endpoint and it seems to be unused.
Does the extension use this endpoint ?
Can we delete it safely ?

The validation script used to check regex aren’t harmful is too strict for restricted contexts regex.
For instance, google is a completely legit restricted context.

To align with the terminology used in the extension

Describe the bug
Please see the Trello card here: https://trello.com/c/vtYQ4WWT/680-quand-je-cr%C3%A9e-enti%C3%A8rement-un-nouveau-contributeur-%C3%A7a-p%C3%A8te-parfois

@JalilArfaoui said Oui problème connu … il faut d’abord crée sans images et ensuite compléter je crois
Etienne said En effet, -> juste le nom -> enregistrer -> ajouter tout le reste
@MaartenLMEM said il faut créer ton contributeur de façon minimaliste et ensuite l'enrchir d'une image etC.

It’s a regression from #88: see this comment.

php bin/console doctrine:migration:migrate --allow-no-migration --no-interaction

I recently added noticesUrls to the contributor normalizer, locally, it's rendered as an array as it should be, but on staging it's rendered as an object with numbered keys :/
Maybe dependencies aren't up to date on staging?

When we’ll release out Aspe and because of #59, the API route will change for /api/v2/ instead of /api/v1/.
cf also dis-moi/extension#88

Hi there,

Great move publishing the code for this service, I can't wait to see it grow!

This issue is a small suggestion to move from GPL to AGPL. This article (in french) shortly summarizes what the difference is. The TLDR is: AGPL also forces the entity that hosts your code on their own servers to share the code if they modify it. Their may be reasons why you would prefer avoiding that, but if you want to add additional legal security to your source code it's a cheap and effective way to do it 🙂

Ce qui désigne le lien secondaire (facultatif), directement vers (en règle général) un produit alternatif

If I just run locally

docker exec kraft-backend_php composer run-script post-install-cmd \
 && docker exec kraft-backend_php bin/console doctrine:migrations:migrate -n --env=test \
 && docker exec kraft-backend_php bin/console cache:clear --env=test \
 && docker exec -e SYMFONY_DEPRECATIONS_HELPER=disabled kraft-backend_php vendor/bin/simple-phpunit

They work 100% each time

But I can reproduce the same error we have in CI if I first

docker-compose down && docker volume prune -f && docker-compose up

It's not random, it’s 100% …

But I can’t see why tests only pass if there has been a previous trial … it seems there is some kind of dependency between test that I’m not seeing … And I don’t even think PHPUnit runs them in parallel …

Originally posted by @JalilArfaoui in #192 (comment)

• Symfony 3.3.x will not receive security fixes. Upgrade to 3.4.x seems reasonable.

• The provisioning (from scratch) of the application from a Docker Compose environment is broken, since the parameters.yml does not exist (and cannot be prompted).

• Then, we’ve got few warnings about deprecated environment variables uses.

• And last but not least, Clever Cloud bucket settings should not be hard-coded (plus staging and production instances should not share the same storage bucket)

• Papertail integration seemed to not work anymore.

The new upcoming interface will give users new sort Feedbacks/Ratings.

Among those, it introduce the counterparts of feedback/rating.
E.g. approve like and dislike have respectively unapprove unlike and undislike counterparts.

Therefore, for each Notice/Recommendation, the total number (must be >= 0) of a ratingType/feedbackType is the difference of a ratingType/feedbackType and its counterparts. E.g. totalNumberOfLikes = numberOfLikes - numberOfUnlikes

So, in API responses...

ratings:
  likes: 10  # ==> 15 likes - 5 unlikes
  dislikes: 3   # ==> 4 dislikes - 1 undislike

RatingTypes should be unforced (so far, it’s pretty much open bar on the v3, and restricted on the v2). Allowed types are:

• approve like
• unapprove unlike
• dislike
• undislike
• display
• detail unfold
• dismiss
• undismiss
• click-message
• click-source
• report

@lilobase
PAR CONTRE : le storage des deux est pour le moment partagé
2:49
(pas la DB, mais l’endoit ou sont stoqué les miniatures)
2:50
donc @narfai regarde si on peut pas connecter vich uploader à un truc style S3
2:50
au lieu de balancer sur le FS

We can connect Vich Uploader to flysystem which provide an FS abstraction with following adapters :

• Local
• Azure
• AWS S3 V2
• AWS S3 V3
• Copy.com
• Dropbox
• FTP
• GridFS
• Memory
• Null / Test
• Rackspace
• ReplicateAdapter
• SFTP
• WebDAV
• PHPCR
• ZipArchive

In order to fit with idempotent clever-cloud deployment, should we provide remote storage parameters through .env file ?

• check out criteria migration (how about seed predefined criteria (see staging backoffice criteria page)
• deploy

• remove protocol fragments in https://github.com/insitu-project/kraft-backend/blob/develop/web/js/regex_validator.js#L23
• change [id$=urlRegex] to [id$="urlRegex"i] in https://github.com/insitu-project/kraft-backend/blob/develop/web/js/regex_validator.js#L43 (case insensitive lookup)

We could use this bundle to force https and allow extended security configuration like HSTS.

Les données ont été entrées de la mauvaise façon en préprod : les données "source" sont sauvegardées en temps que "alternative". Transférer les données de la table source vers la table alternative.

Migrations are finally run after deployment. They hardly fail though...

2018-11-23T15:44:22+01:00 Running CC_POST_BUILD_HOOK: ./clevercloud/post_build.sh
2018-11-23T15:44:23+01:00 Application Migrations
2018-11-23T15:44:23+01:00
2018-11-23T15:44:23+01:00 WARNING! You have 2 previously executed migrations in the database that are not registered migrations.
2018-11-23T15:44:23+01:00
2018-11-23T15:44:23+01:00 >> 2017-05-26 13:10:58 (20170526131058)
2018-11-23T15:44:23+01:00 >> 2017-05-26 14:27:17 (20170526142717)
2018-11-23T15:44:23+01:00 Migrating up to 20181026153141 from 20161214123901
2018-11-23T15:44:23+01:00 Migration 20181025174235 failed during Execution. Error An exception occurred while executing 'ALTER TABLE fos_user DROP locked, DROP expired, DROP expires_at, DROP credentials_expired, DROP credentials_expire_at, CHANGE username username VARCHAR(180) NOT NULL, CHANGE username_canonical username_canonical VARCHAR(180) NOT NULL, CHANGE email email VARCHAR(180) NOT NULL, CHANGE email_canonical email_canonical VARCHAR(180) NOT NULL, CHANGE salt salt VARCHAR(255) DEFAULT NULL, CHANGE last_login last_login DATETIME DEFAULT NULL, CHANGE confirmation_token confirmation_token VARCHAR(180) DEFAULT NULL, CHANGE password_requested_at password_requested_at DATETIME DEFAULT NULL':
2018-11-23T15:44:23+01:00 -> ALTER TABLE fos_user DROP locked, DROP expired, DROP expires_at, DROP credentials_expired, DROP credentials_expire_at, CHANGE username username VARCHAR(180) NOT NULL, CHANGE username_canonical username_canonical VARCHAR(180) NOT NULL, CHANGE email email VARCHAR(180) NOT NULL, CHANGE email_canonical email_canonical VARCHAR(180) NOT NULL, CHANGE salt salt VARCHAR(255) DEFAULT NULL, CHANGE last_login last_login DATETIME DEFAULT NULL, CHANGE confirmation_token confirmation_token VARCHAR(180) DEFAULT NULL, CHANGE password_requested_at password_requested_at DATETIME DEFAULT NULL
2018-11-23T15:44:23+01:00 ++ migrating 20181025174235
2018-11-23T15:44:23+01:00 SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'locked'; check that column/key exists
2018-11-23T15:44:24+01:00 SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'locked'; check that column/key exists" ["exception" => Doctrine\DBAL\Exception\DriverException { …},"command" => "'doctrine:migration:migrate' --allow-no-migration --no-interaction","message" => """ An exception occurred while executing 'ALTER TABLE fos_user DROP locked, DROP expired, DROP expires_at, DROP credentials_expired, DROP credentials_expire_at, CHANGE username username VARCHAR(180) NOT NULL, CHANGE username_canonical username_canonical VARCHAR(180) NOT NULL, CHANGE email email VARCHAR(180) NOT NULL, CHANGE email_canonical email_canonical VARCHAR(180) NOT NULL, CHANGE salt salt VARCHAR(255) DEFAULT NULL, CHANGE last_login last_login DATETIME DEFAULT NULL, CHANGE confirmation_token confirmation_token VARCHAR(180) DEFAULT NULL, CHANGE password_requested_at password_requested_at DATETIME DEFAULT NULL':\n \n SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'locked'; check that column/key exists """]
2018-11-23T15:44:24+01:00 14:44:23 ERROR [console] Error thrown while running command "'doctrine:migration:migrate' --allow-no-migration --no-interaction". Message: "An exception occurred while executing 'ALTER TABLE fos_user DROP locked, DROP expired, DROP expires_at, DROP credentials_expired, DROP credentials_expire_at, CHANGE username username VARCHAR(180) NOT NULL, CHANGE username_canonical username_canonical VARCHAR(180) NOT NULL, CHANGE email email VARCHAR(180) NOT NULL, CHANGE email_canonical email_canonical VARCHAR(180) NOT NULL, CHANGE salt salt VARCHAR(255) DEFAULT NULL, CHANGE last_login last_login DATETIME DEFAULT NULL, CHANGE confirmation_token confirmation_token VARCHAR(180) DEFAULT NULL, CHANGE password_requested_at password_requested_at DATETIME DEFAULT NULL':
2018-11-23T15:44:26+01:00 An exception occurred while executing 'ALTER TABLE fos_user DROP locked, DR
2018-11-23T15:44:26+01:00 In AbstractMySQLDriver.php line 126:
2018-11-23T15:44:26+01:00
2018-11-23T15:44:26+01:00 (180) NOT NULL, CHANGE email_canonical email_canonical VARCHAR(180) NOT NUL
2018-11-23T15:44:26+01:00 onical username_canonical VARCHAR(180) NOT NULL, CHANGE email email VARCHAR
2018-11-23T15:44:26+01:00 ire_at, CHANGE username username VARCHAR(180) NOT NULL, CHANGE username_can
2018-11-23T15:44:26+01:00 OP expired, DROP expires_at, DROP credentials_expired, DROP credentials_exp
2018-11-23T15:44:26+01:00 SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'locked'
2018-11-23T15:44:26+01:00
2018-11-23T15:44:26+01:00 ETIME DEFAULT NULL':
2018-11-23T15:44:26+01:00 R(180) DEFAULT NULL, CHANGE password_requested_at password_requested_at DAT
2018-11-23T15:44:26+01:00 DATETIME DEFAULT NULL, CHANGE confirmation_token confirmation_token VARCHA
2018-11-23T15:44:26+01:00 L, CHANGE salt salt VARCHAR(255) DEFAULT NULL, CHANGE last_login last_login
2018-11-23T15:44:26+01:00 ; check that column/key exists
2018-11-23T15:44:26+01:00
2018-11-23T15:44:26+01:00 In PDOConnection.php line 109:
2018-11-23T15:44:26+01:00
2018-11-23T15:44:26+01:00
2018-11-23T15:44:26+01:00 SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'locked'
2018-11-23T15:44:26+01:00 SQLSTATE[42000]: Syntax error or access violation: 1091 Can't DROP 'locked'
2018-11-23T15:44:26+01:00 ; check that column/key exists
2018-11-23T15:44:26+01:00
2018-11-23T15:44:26+01:00 In PDOConnection.php line 107:
2018-11-23T15:44:26+01:00 DIGEST-MD5 common mech free
2018-11-23T15:44:26+01:00 POST_BUILD_HOOK failed, aborting
2018-11-23T15:44:26+01:00 ; check that column/key exists 

From the API, the contributors’ avatar should be a full URL.

Benjamin Menant
17:48 Au passage, vous préférez quoi, pour consommer une API:

{
  boby: {
    image: 'avatar-boby.jpg',
    //...
  }
}

ou quelque chose comme

{
  boby: {
    image: 'https://alotofbobs.io/uploads/avatar/avatar-boby.jpg',
    //...
  }
}

? (edited)

Arnaud LEMAIRE
17:49 perso je préfère le 2 car tu rend plus facile les évolutions
17:49 sans avoir besoin de mettre à jour le consommateur

Not having a contributor in a recommendation make the api call http://localhost:8080/api/v1/recommendation/1 crash.
Make the field mandatory

Context
We have more and more external contributors that uses BO.
"Utilisateur" status being not working, we give them amdin status. Thus, they can do everything, such as other user passwords !

Aim
We want to give them a normal user role with restricted rights to access and modify some parts.

Currently, there is a User Profil.
Open him access to

• "Bulles" section
• "Domaines" section
  Consequently, User has NOT acess to "Utilisateurs" section, nor "Groupes de domaines".

Not sure the best way to describe this but wanted to get it written down - looking for your help @JalilArfaoui @achauve !
...

payload actuel :

{
contributor: {
  image: "http://localhost:8080/uploads/avatars/photo.jpg",
  name: "John Doe",
  organization: "LMEM"
},
visibility: "public",
title: "Un site de new avec des info fiables",
description: "Il torche sa maman",
alternatives: [ ],
resource: {
  author: "",
  url: "",
  label: ""
},
criteria: [ ]
}

Soit on rajoute un niveau dans resource

resource: {
  author: "",
  url: "",
  label: "",
  editor: {
    name: "",
    url: ""
  }
},

Soit on applati (pas tres resty)

resource: {
  author: "",
  url: "",
  label: "",
  editor_name: name: "",
  editor_url: url: "",
  }
},

@bmenant opinions ? (avant de pinger le reste des gens concernés)

https://preprod-lmem-craft-backend.cleverapps.io/api/v2/recommendation/3 returns:

{"error":{"code":500,"message":"Internal Server Error"}}

Ce qui désigne le lien principal proposé par l'extension (typiquement, un article, un comparatif, un guide d'achat ...)

I think that it would be nice to be able to self-host the backend of DisMoi and to be able to retrieve information from multiple backend. I know that it would require a huge work, but I think that it could help to spread the use of this awesome project as any organization could host its own information "aggregator" and "backend".

It could maybe be done by using the activitypub protocol to federate the backend as it could allow to interact with other application using this protocol.

Currently, https://preprod-lmem-craft-backend.cleverapps.io/api/v1/matchingcontexts returns :

[{"recommendation_url":"http:\/\/preprod-lmem-craft-backend.cleverapps.io\/api\/v1\/recommendation\/3","url_regex":"20minutes.fr\/societe\/1860259-20160606-violences-policieres-quarante-huit-enquetes-judiciaires-ouvertes-police-polices"},{"recommendation_url":"http:\/\/preprod-lmem-craft-backend.cleverapps.io\/api\/v1\/recommendation\/6","url_regex":"http:\/\/tests.menant-benjamin.fr\/?$"},{"recommendation_url":"http:\/\/preprod-lmem-craft-backend.cleverapps.io\/api\/v1\/recommendation\/9","url_regex":"http:\/\/www.lactalis.fr\/le-groupe\/histoire\/"},{"recommendation_url":"http:\/\/preprod-lmem-craft-backend.cleverapps.io\/api\/v1\/recommendation\/9","url_regex":"http:\/\/www.lactalis.fr\/le-groupe\/chiffres-cles\/#\/ca"},{"recommendation_url":"http:\/\/preprod-lmem-craft-backend.cleverapps.io\/api\/v1\/recommendation\/10","url_regex":"http:\/\/www.bonbonsgourmands.fr\/"},{"recommendation_url":"http:\/\/preprod-lmem-craft-backend.cleverapps.io\/api\/v1\/recommendation\/11","url_regex":"http:\/\/www.fnac.com\/mp23768793\/Wiko-Highway-Signs-violet-3G-HSPA-8-Go-GSM-Android-smartphone\/w-4"}]

The recommendation_url field contains http urls. It should be https urls, otherwise it forces to add a new CSP exception in the manifest while there is no good reason (especially given the time was spent to have proper HTTPS support even in staging :-p).
This is a block for dis-moi/extension#59 (it could be unblocked by changing the CSP but a backend change would be more appropriate)

On the Recommandation list, write script tag with unhandled type (json/text) which contains json-encoded MatchingContexts associated to the current user's private recommendations.
If the current user has Editor role, we have to display MatchingContexts of the whole Editor's Organization.
This script tag will be retrieved by extensions for pre-visualization.

For example https://github.com/FriendsOfPHP/PHP-CS-Fixer

Hi @lilobase ! I deployed the backend and i get a ParameterNotFoundException when i try to access to http://localhost:8000 after running local server with "nf run php bin/console server:run".
The "nf run php ./composer.phar install" step runs well with a .env file which fit to my local mysql configuration.
My app/config/parameters.yml file looks fine and i also force the variables_order parameter to "EGPCS" in my php.ini file.
Should i replace the node-foreman implementation (nf command above) by the original one ?

ParameterNotFoundException in ParameterBag.php line 84: You have requested a non-existent parameter "database.host". Did you mean one of these: "database_host", "database_port"?