Merge branch 'develop' of git.pleroma.social:pleroma/pleroma into snoot

.gitignore

@@ -6,6 +6,9 @@
 /uploads
 /test/uploads
 /.elixir_ls
+/test/fixtures/test_tmp.txt
+/test/fixtures/image_tmp.jpg
+/doc
 
 # Prevent committing custom emojis
 /priv/static/emoji/custom/*
@@ -28,4 +31,4 @@ erl_crash.dump
 .env
 
 # Editor config
-/.vscode
+/.vscode

.gitlab-ci.yml

@@ -1,4 +1,4 @@
-image: elixir:1.6.4
+image: elixir:1.7.2
 
 services:
   - postgres:9.6.2
@@ -9,6 +9,11 @@ variables:
   POSTGRES_PASSWORD: postgres
   DB_HOST: postgres
 
+cache:
+  key: ${CI_COMMIT_REF_SLUG}
+  paths:
+          - deps
+          - _build
 stages:
   - lint
   - test

CONFIGURATION.md

@@ -1,106 +0,0 @@
-# Configuring Pleroma
-
-In the `config/` directory, you will find the following relevant files:
-
-* `config.exs`: default base configuration
-* `dev.exs`: default additional configuration for `MIX_ENV=dev`
-* `prod.exs`: default additional configuration for `MIX_ENV=prod`
-
-
-Do not modify files in the list above.
-Instead, overload the settings by editing the following files:
-
-* `dev.secret.exs`: custom additional configuration for `MIX_ENV=dev`
-* `prod.secret.exs`: custom additional configuration for `MIX_ENV=prod`
-
-## Uploads configuration
-
-To configure where to upload files, and wether or not 
-you want to remove automatically EXIF data from pictures
-being uploaded.
-
-    config :pleroma, Pleroma.Upload,
-      uploads: "uploads",
-      strip_exif: false
-
-* `uploads`: where to put the uploaded files, relative to pleroma's main directory.
-* `strip_exif`: whether or not to remove EXIF data from uploaded pics automatically. 
-   This needs Imagemagick installed on the system ( apt install imagemagick ).
-
-
-## Block functionality
-
-    config :pleroma, :activitypub,
-      accept_blocks: true,
-      unfollow_blocked: true,
-      outgoing_blocks: true
-
-    config :pleroma, :user, deny_follow_blocked: true
-
-* `accept_blocks`: whether to accept incoming block activities from
-   other instances
-* `unfollow_blocked`: whether blocks result in people getting
-   unfollowed
-* `outgoing_blocks`: whether to federate blocks to other instances
-* `deny_follow_blocked`: whether to disallow following an account that
-   has blocked the user in question
-
-## Message Rewrite Filters (MRFs)
-
-Modify incoming and outgoing posts.
-
-    config :pleroma, :instance,
-      rewrite_policy: Pleroma.Web.ActivityPub.MRF.NoOpPolicy
-
-`rewrite_policy` specifies which MRF policies to apply.
-It can either be a single policy or a list of policies.
-Currently, MRFs availible by default are:
-
-* `Pleroma.Web.ActivityPub.MRF.NoOpPolicy`
-* `Pleroma.Web.ActivityPub.MRF.DropPolicy`
-* `Pleroma.Web.ActivityPub.MRF.SimplePolicy`
-* `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`
-
-Some policies, such as SimplePolicy and RejectNonPublic,
-can be additionally configured in their respective sections.
-
-### NoOpPolicy
-
-Does not modify posts (this is the default `rewrite_policy`)
-
-### DropPolicy
-
-Drops all posts.
-It generally does not make sense to use this in production.
-
-### SimplePolicy
-
-Restricts the visibility of posts from certain instances.
-
-    config :pleroma, :mrf_simple,
-      media_removal: [],
-      media_nsfw: [],
-      federated_timeline_removal: [],
-      reject: [],
-      accept: []
-
-* `media_removal`: posts from these instances will have attachments 
-   removed
-* `media_nsfw`: posts from these instances will have attachments marked
-   as nsfw
-* `federated_timeline_removal`: posts from these instances will be 
-   marked as unlisted
-* `reject`: posts from these instances will be dropped
-* `accept`: if not empty, only posts from these instances will be accepted
-
-### RejectNonPublic
-
-Drops posts with non-public visibility settings.
-
-    config :pleroma :mrf_rejectnonpublic
-      allow_followersonly: false,
-      allow_direct: false,
-
-* `allow_followersonly`: whether to allow follower-only posts through
-   the filter
-* `allow_direct`: whether to allow direct messages through the filter

README.md

@@ -2,11 +2,13 @@
 
 ## About Pleroma
 
-Pleroma is an OStatus-compatible social networking server written in Elixir, compatible with GNU Social and Mastodon. It is high-performance and can run on small devices like a Raspberry Pi.
+Pleroma is a microblogging server software that can federate (= exchange messages with) other servers that support the same federation standards (OStatus and ActivityPub). What that means is that you can host a server for yourself or your friends and stay in control of your online identity, but still exchange messages with people on larger servers. Pleroma will federate with all servers that implement either OStatus or ActivityPub, like Friendica, GNU Social, Hubzilla, Mastodon, Misskey, Peertube, and Pixelfed.
+
+Pleroma is written in Elixir, high-performance and can run on small devices like a Raspberry Pi.
 
 For clients it supports both the [GNU Social API with Qvitter extensions](https://twitter-api.readthedocs.io/en/latest/index.html) and the [Mastodon client API](https://github.com/tootsuite/documentation/blob/master/Using-the-API/API.md).
 
-Mobile clients that are known to work well:
+Client applications that are known to work well:
 
 * Twidere
 * Tusky
@@ -15,6 +17,7 @@ Mobile clients that are known to work well:
 * Amaroq (iOS)
 * Tootdon (Android + iOS)
 * Tootle (iOS)
+* Whalebird (Windows + Mac + Linux)
 
 No release has been made yet, but several servers have been online for months already. If you want to run your own server, feel free to contact us at @lain@pleroma.soykaf.com or in our dev chat at #pleroma on freenode or via matrix at https://matrix.heldscal.la/#/room/#freenode_#pleroma:matrix.org.
 
@@ -22,7 +25,7 @@ No release has been made yet, but several servers have been online for months al
 
 ### Docker
 
-While we don't provide docker files, other people have written very good ones. Take a look at https://github.com/Angristan/dockerfiles/tree/master/pleroma or https://github.com/sn0w/pleroma-docker.
+While we don't provide docker files, other people have written very good ones. Take a look at https://github.com/angristan/docker-pleroma or https://github.com/sn0w/pleroma-docker.
 
 ### Dependencies
 
@@ -34,18 +37,15 @@ While we don't provide docker files, other people have written very good ones. T
 
   * Run `mix deps.get` to install elixir dependencies.
 
-  * Run `mix generate_config`. This will ask you a few questions about your instance and generate a configuration file in `config/generated_config.exs`. Check that and copy it to either `config/dev.secret.exs` or `config/prod.secret.exs`. It will also create a `config/setup_db.psql`; you may want to double-check this file in case you wanted a different username, or database name than the default. Then you need to run the script as PostgreSQL superuser (i.e. `sudo su postgres -c "psql -f config/setup_db.psql"`). It will create a pleroma db user, database and will setup needed extensions that need to be set up. Postgresql super-user privileges are only needed for this step.
+  * Run `mix pleroma.instance gen`. This will ask you questions about your instance and generate a configuration file in `config/generated_config.exs`. Check that and copy it to either `config/dev.secret.exs` or `config/prod.secret.exs`. It will also create a `config/setup_db.psql`, which you should run as the PostgreSQL superuser (i.e., `sudo -u postgres psql -f config/setup_db.psql`). It will create the database, user, and password you gave `mix pleroma.gen.instance` earlier, as well as set up the necessary extensions in the database. PostgreSQL superuser privileges are only needed for this step.
 
-  * For these next steps, the default will be to run pleroma using the dev configuration file, `config/dev.secret.exs`. To run them using the prod config file, prefix each command at the shell with `MIX_ENV=prod`. For example: `MIX_ENV=prod mix phx.server`.
+  * For these next steps, the default will be to run pleroma using the dev configuration file, `config/dev.secret.exs`. To run them using the prod config file, prefix each command at the shell with `MIX_ENV=prod`. For example: `MIX_ENV=prod mix phx.server`. Documentation for the config can be found at [``config/config.md``](config/config.md)
 
   * Run `mix ecto.migrate` to run the database migrations. You will have to do this again after certain updates.
 
   * You can check if your instance is configured correctly by running it with `mix phx.server` and checking the instance info endpoint at `/api/v1/instance`. If it shows your uri, name and email correctly, you are configured correctly. If it shows something like `localhost:4000`, your configuration is probably wrong, unless you are running a local development setup.
 
-  * The common and convenient way for adding HTTPS is by using Nginx as a reverse proxy. You can look at example Nginx configuration in `installation/pleroma.nginx`. If you need TLS/SSL certificates for HTTPS, you can look get some for free with letsencrypt: https://letsencrypt.org/
-  The simplest way to obtain and install a certificate is to use [Certbot.](https://certbot.eff.org) Depending on your specific setup, certbot may be able to get a certificate and configure your web server automatically.
-
-  * [Not tested with system reboot yet!] You'll also want to set up Pleroma to be run as a systemd service. Example .service file can be found in `installation/pleroma.service` you can put it in `/etc/systemd/system/`.
+  * The common and convenient way for adding HTTPS is by using Nginx as a reverse proxy. You can look at example Nginx configuration in `installation/pleroma.nginx`. If you need TLS/SSL certificates for HTTPS, you can look get some for free with letsencrypt: <https://letsencrypt.org/>. The simplest way to obtain and install a certificate is to use [Certbot.](https://certbot.eff.org) Depending on your specific setup, certbot may be able to get a certificate and configure your web server automatically.
 
 ## Running
 
@@ -55,9 +55,15 @@ While we don't provide docker files, other people have written very good ones. T
 Pleroma comes with two frontends. The first one, Pleroma FE, can be reached by normally visiting the site. The other one, based on the Mastodon project, can be found by visiting the /web path of your site.
 
 ### As systemd service (with provided .service file)
+Example .service file can be found in `installation/pleroma.service` you can put it in `/etc/systemd/system/`.
 Running `service pleroma start`
 Logs can be watched by using `journalctl -fu pleroma.service`
 
+### As OpenRC service (with provided RC file)
+Copy ``installation/init.d/pleroma`` to ``/etc/init.d/pleroma``.
+You can add it to the services ran by default with:
+``rc-update add pleroma``
+
 ### Standalone/run by other means
 Run `mix phx.server` in repository's root, it will output log into stdout/stderr
 
@@ -70,22 +76,6 @@ Add the following to your `dev.secret.exs` or `prod.secret.exs` if you want to p
 
 This is useful for running pleroma inside Tor or i2p.
 
-## Admin Tasks
-
-### Register a User
-
-Run `mix register_user <name> <nickname> <email> <bio> <password>`. The `name` appears on statuses, while the nickname corresponds to the user, e.g. `@nickname@instance.tld`
-
-### Password reset
-
-Run `mix generate_password_reset username` to generate a password reset link that you can then send to the user.
-
-### Moderators
-
-You can make users moderators. They will then be able to delete any post.
-
-Run `mix set_moderator username [true|false]` to make user a moderator or not.
-
 ## Troubleshooting
 
 ### No incoming federation

config/config.exs

@@ -10,27 +10,49 @@ config :pleroma, ecto_repos: [Pleroma.Repo]
 
 config :pleroma, Pleroma.Repo, types: Pleroma.PostgresTypes
 
+# Upload configuration
 config :pleroma, Pleroma.Upload,
   uploader: Pleroma.Uploaders.Local,
-  strip_exif: false
+  filters: [],
+  proxy_remote: false,
+  proxy_opts: []
 
-config :pleroma, Pleroma.Uploaders.Local,
-  uploads: "uploads",
-  uploads_url: "{{base_url}}/media/{{file}}"
+config :pleroma, Pleroma.Uploaders.Local, uploads: "uploads"
 
 config :pleroma, Pleroma.Uploaders.S3,
   bucket: nil,
   public_endpoint: "https://s3.amazonaws.com"
 
+config :pleroma, Pleroma.Uploaders.MDII,
+  cgi: "https://mdii.sakura.ne.jp/mdii-post.cgi",
+  files: "https://mdii.sakura.ne.jp"
+
 config :pleroma, :emoji, shortcode_globs: ["/emoji/custom/**/*.png"]
 
-config :pleroma, :uri_schemes, additionnal_schemes: []
+config :pleroma, :uri_schemes,
+  valid_schemes: [
+    "https",
+    "http",
+    "dat",
+    "dweb",
+    "gopher",
+    "ipfs",
+    "ipns",
+    "irc",
+    "ircs",
+    "magnet",
+    "mailto",
+    "mumble",
+    "ssb",
+    "xmpp"
+  ]
 
 # Configures the endpoint
 config :pleroma, Pleroma.Web.Endpoint,
   url: [host: "localhost"],
   protocol: "https",
   secret_key_base: "aK4Abxf29xU9TTDKre9coZPUgevcVCFQJe/5xP/7Lt4BEif6idBIbjupVbOrbKxl",
+  signing_salt: "CqaoopA2",
   render_errors: [view: Pleroma.Web.ErrorView, accepts: ~w(json)],
   pubsub: [name: Pleroma.PubSub, adapter: Phoenix.PubSub.PG2],
   secure_cookie_flag: true
@@ -50,31 +72,34 @@ config :mime, :types, %{
 config :pleroma, :websub, Pleroma.Web.Websub
 config :pleroma, :ostatus, Pleroma.Web.OStatus
 config :pleroma, :httpoison, Pleroma.HTTP
-
-version =
-  with {version, 0} <- System.cmd("git", ["rev-parse", "HEAD"]) do
-    "Pleroma #{Mix.Project.config()[:version]} #{String.trim(version)}"
-  else
-    _ -> "Pleroma #{Mix.Project.config()[:version]} dev"
-  end
+config :tesla, adapter: Tesla.Adapter.Hackney
 
 # Configures http settings, upstream proxy etc.
 config :pleroma, :http, proxy_url: nil
 
 config :pleroma, :instance,
-  version: version,
   name: "Pleroma",
   email: "example@example.com",
   description: "A Pleroma instance, an alternative fediverse server",
   limit: 5000,
   upload_limit: 16_000_000,
+  avatar_upload_limit: 2_000_000,
+  background_upload_limit: 4_000_000,
+  banner_upload_limit: 4_000_000,
   registrations_open: true,
   federating: true,
   allow_relay: true,
   rewrite_policy: Pleroma.Web.ActivityPub.MRF.NoOpPolicy,
   public: true,
   quarantined_instances: [],
-  managed_config: true
+  managed_config: true,
+  allowed_post_formats: [
+    "text/plain",
+    "text/html",
+    "text/markdown"
+  ],
+  finmoji_enabled: true,
+  mrf_transparency: true
 
 config :pleroma, :markup,
   # XXX - unfortunately, inline images must be enabled by default right now, because
@@ -83,7 +108,10 @@ config :pleroma, :markup,
   allow_headings: false,
   allow_tables: false,
   allow_fonts: false,
-  scrub_policy: Pleroma.HTML.Scrubber.Default
+  scrub_policy: [
+    Pleroma.HTML.Transform.MediaProxy,
+    Pleroma.HTML.Scrubber.Default
+  ]
 
 config :pleroma, :fe,
   theme: "pleroma-dark",
@@ -95,15 +123,24 @@ config :pleroma, :fe,
   redirect_root_login: "/main/friends",
   show_instance_panel: true,
   scope_options_enabled: false,
-  collapse_message_with_subject: false
+  formatting_options_enabled: false,
+  collapse_message_with_subject: false,
+  hide_post_stats: false,
+  hide_user_stats: false,
+  scope_copy: true,
+  subject_line_behavior: "email",
+  always_show_subject_input: true
 
 config :pleroma, :activitypub,
   accept_blocks: true,
   unfollow_blocked: true,
-  outgoing_blocks: true
+  outgoing_blocks: true,
+  follow_handshake_timeout: 500
 
 config :pleroma, :user, deny_follow_blocked: true
 
+config :pleroma, :mrf_normalize_markup, scrub_policy: Pleroma.HTML.Scrubber.Default
+
 config :pleroma, :mrf_rejectnonpublic,
   allow_followersonly: false,
   allow_direct: false
@@ -117,9 +154,11 @@ config :pleroma, :mrf_simple,
 
 config :pleroma, :media_proxy,
   enabled: false,
-  redirect_on_failure: true
-
-# base_url: "https://cache.pleroma.social"
+  # base_url: "https://cache.pleroma.social",
+  proxy_opts: [
+    # inline_content_types: [] | false | true,
+    # http: [:insecure]
+  ]
 
 config :pleroma, :chat, enabled: true
 
@@ -140,6 +179,27 @@ config :pleroma, :suggestions,
   limit: 23,
   web: "https://vinayaka.distsn.org/?{{host}}+{{user}}"
 
+config :pleroma, :http_security,
+  enabled: true,
+  sts: false,
+  sts_max_age: 31_536_000,
+  ct_max_age: 2_592_000,
+  referrer_policy: "same-origin"
+
+config :cors_plug,
+  max_age: 86_400,
+  methods: ["POST", "PUT", "DELETE", "GET", "PATCH", "OPTIONS"],
+  expose: [
+    "Link",
+    "X-RateLimit-Reset",
+    "X-RateLimit-Limit",
+    "X-RateLimit-Remaining",
+    "X-Request-Id",
+    "Idempotency-Key"
+  ],
+  credentials: true,
+  headers: ["Authorization", "Content-Type", "Idempotency-Key"]
+
 # Import environment specific config. This must remain at the bottom
 # of this file so it overrides the configuration defined above.
 import_config "#{Mix.env()}.exs"

config/config.md

@@ -0,0 +1,129 @@
+# Configuration
+
+This file describe the configuration, it is recommended to edit the relevant *.secret.exs file instead of the others founds in the ``config`` directory.
+If you run Pleroma with ``MIX_ENV=prod`` the file is ``prod.secret.exs``, otherwise it is ``dev.secret.exs``.
+
+## Pleroma.Upload
+* `uploader`: Select which `Pleroma.Uploaders` to use
+* `filters`: List of `Pleroma.Upload.Filter` to use.
+* `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host.
+* `proxy_remote`: If you're using a remote uploader, Pleroma will proxy media requests instead of redirecting to it.
+* `proxy_opts`: Proxy options, see `Pleroma.ReverseProxy` documentation.
+
+Note: `strip_exif` has been replaced by `Pleroma.Upload.Filter.Mogrify`.
+
+## Pleroma.Uploaders.Local
+* `uploads`: Which directory to store the user-uploads in, relative to pleroma’s working directory
+
+## Pleroma.Upload.Filter.Mogrify
+
+* `args`: List of actions for the `mogrify` command like `"strip"` or `["strip", {"impode", "1"}]`.
+
+## Pleroma.Upload.Filter.Dedupe
+
+No specific configuration.
+
+## Pleroma.Upload.Filter.AnonymizeFilename
+
+This filter replaces the filename (not the path) of an upload. For complete obfuscation, add
+`Pleroma.Upload.Filter.Dedupe` before AnonymizeFilename.
+
+* `text`: Text to replace filenames in links. If empty, `{random}.extension` will be used.
+
+## :uri_schemes
+* `valid_schemes`: List of the scheme part that is considered valid to be an URL
+
+## :instance
+* `name`: The instance’s name
+* `email`: Email used to reach an Administrator/Moderator of the instance
+* `description`: The instance’s description, can be seen in nodeinfo and ``/api/v1/instance``
+* `limit`: Posts character limit (CW/Subject included in the counter)
+* `upload_limit`: File size limit of uploads (except for avatar, background, banner)
+* `avatar_upload_limit`: File size limit of user’s profile avatars
+* `background_upload_limit`: File size limit of user’s profile backgrounds
+* `banner_upload_limit`: File size limit of user’s profile banners
+* `registrations_open`: Enable registrations for anyone, invitations can be used when false.
+* `federating`: Enable federation with other instances
+* `allow_relay`: Enable Pleroma’s Relay, which makes it possible to follow a whole instance
+* `rewrite_policy`: Message Rewrite Policy, either one or a list. Here are the ones available by default:
+  * `Pleroma.Web.ActivityPub.MRF.NoOpPolicy`: Doesn’t modify activities (default)
+  * `Pleroma.Web.ActivityPub.MRF.DropPolicy`: Drops all activities. It generally doesn’t makes sense to use in production
+  * `Pleroma.Web.ActivityPub.MRF.SimplePolicy`: Restrict the visibility of activities from certains instances (See ``:mrf_simple`` section)
+  * `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`: Drops posts with non-public visibility settings (See ``:mrf_rejectnonpublic`` section)
+  * `Pleroma.Web.ActivityPub.MRF.EnsureRePrepended`: Rewrites posts to ensure that replies to posts with subjects do not have an identical subject and instead begin with re:.
+* `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
+* `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
+* `managed_config`: Whenether the config for pleroma-fe is configured in this config or in ``static/config.json``
+* `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML)
+* `finmoji_enabled`: Whenether to enable the finmojis in the custom emojis.
+* `mrf_transparency`: Make the content of your Message Rewrite Facility settings public (via nodeinfo).
+* `scope_copy`: Copy the scope (private/unlisted/public) in replies to posts by default.
+* `subject_line_behavior`: Allows changing the default behaviour of subject lines in replies. Valid values:
+  * "email": Copy and preprend re:, as in email.
+  * "masto": Copy verbatim, as in Mastodon.
+  * "noop": Don't copy the subject.
+* `always_show_subject_input`: When set to false, auto-hide the subject field when it's empty.
+
+## :fe
+This section is used to configure Pleroma-FE, unless ``:managed_config`` in ``:instance`` is set to false.
+
+* `theme`: Which theme to use, they are defined in ``styles.json``
+* `logo`: URL of the logo, defaults to Pleroma’s logo
+* `logo_mask`: Whenether to mask the logo
+* `logo_margin`: What margin to use around the logo
+* `background`: URL of the background, unless viewing a user profile with a background that is set
+* `redirect_root_no_login`: relative URL which indicates where to redirect when a user isn’t logged in.
+* `redirect_root_login`: relative URL which indicates where to redirect when a user is logged in.
+* `show_instance_panel`: Whenether to show the instance’s specific panel.
+* `scope_options_enabled`: Enable setting an notice visibility and subject/CW when posting
+* `formatting_options_enabled`: Enable setting a formatting different than plain-text (ie. HTML, Markdown) when posting, relates to ``:instance, allowed_post_formats``
+* `collapse_message_with_subjects`: When a message has a subject(aka Content Warning), collapse it by default
+* `hide_post_stats`: Hide notices statistics(repeats, favorites, …)
+* `hide_user_stats`: Hide profile statistics(posts, posts per day, followers, followings, …)
+
+## :mrf_simple
+* `media_removal`: List of instances to remove medias from
+* `media_nsfw`: List of instances to put medias as NSFW(sensitive) from
+* `federated_timeline_removal`: List of instances to remove from Federated (aka The Whole Known Network) Timeline
+* `reject`: List of instances to reject any activities from
+* `accept`: List of instances to accept any activities from
+
+## :mrf_rejectnonpublic
+* `allow_followersonly`: whether to allow followers-only posts
+* `allow_direct`: whether to allow direct messages
+
+## :media_proxy
+* `enabled`: Enables proxying of remote media to the instance’s proxy
+* `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.
+* `proxy_opts`: All options defined in `Pleroma.ReverseProxy` documentation, defaults to `[max_body_length: (25*1_048_576)]`.
+
+## :gopher
+* `enabled`: Enables the gopher interface
+* `ip`: IP address to bind to
+* `port`: Port to bind to
+
+## :activitypub
+* ``accept_blocks``: Whether to accept incoming block activities from other instances
+* ``unfollow_blocked``: Whether blocks result in people getting unfollowed
+* ``outgoing_blocks``: Whether to federate blocks to other instances
+* ``deny_follow_blocked``: Whether to disallow following an account that has blocked the user in question
+
+## :http_security
+* ``enabled``: Whether the managed content security policy is enabled
+* ``sts``: Whether to additionally send a `Strict-Transport-Security` header
+* ``sts_max_age``: The maximum age for the `Strict-Transport-Security` header if sent
+* ``ct_max_age``: The maximum age for the `Expect-CT` header if sent
+* ``referrer_policy``: The referrer policy to use, either `"same-origin"` or `"no-referrer"`.
+
+## :mrf_user_allowlist
+
+The keys in this section are the domain names that the policy should apply to.
+Each key should be assigned a list of users that should be allowed through by
+their ActivityPub ID.
+
+An example:
+
+```
+config :pleroma, :mrf_user_allowlist,
+  "example.org": ["https://example.org/users/admin"]
+```

config/dev.exs

@@ -49,11 +49,10 @@ config :pleroma, Pleroma.Repo,
   hostname: "localhost",
   pool_size: 10
 
-try do
+if File.exists?("./config/dev.secret.exs") do
   import_config "dev.secret.exs"
-rescue
-  _ ->
-    IO.puts(
-      "!!! RUNNING IN LOCALHOST DEV MODE! !!!\nFEDERATION WON'T WORK UNTIL YOU CONFIGURE A dev.secret.exs"
-    )
+else
+  IO.puts(
+    "!!! RUNNING IN LOCALHOST DEV MODE! !!!\nFEDERATION WON'T WORK UNTIL YOU CONFIGURE A dev.secret.exs"
+  )
 end

config/test.exs

@@ -9,7 +9,7 @@ config :pleroma, Pleroma.Web.Endpoint,
 # Print only warnings and errors during test
 config :logger, level: :warn
 
-config :pleroma, Pleroma.Upload, uploads: "test/uploads"
+config :pleroma, Pleroma.Uploaders.Local, uploads: "test/uploads"
 
 # Configure your database
 config :pleroma, Pleroma.Repo,
@@ -25,7 +25,13 @@ config :pbkdf2_elixir, rounds: 1
 
 config :pleroma, :websub, Pleroma.Web.WebsubMock
 config :pleroma, :ostatus, Pleroma.Web.OStatusMock
-config :pleroma, :httpoison, HTTPoisonMock
+config :tesla, adapter: Tesla.Mock
+
+config :web_push_encryption, :vapid_details,
+  subject: "mailto:administrator@example.com",
+  public_key:
+    "BLH1qVhJItRGCfxgTtONfsOKDc9VRAraXw-3NsmjMngWSh7NxOizN6bkuRA7iLTMPS82PjwJAr3UoK9EC1IFrz4",
+  private_key: "_-XZ0iebPrRfZ_o0-IatTdszYa8VCH1yLN-JauK7HHA"
 
 try do
   import_config "test.secret.exs"

installation/caddyfile-pleroma.example

@@ -1,4 +1,10 @@
-social.domain.tld  {
+# default Caddyfile config for Pleroma
+#
+# Simple installation instructions:
+# 1. Replace 'example.tld' with your instance's domain wherever it appears.
+# 2. Copy this section into your Caddyfile and restart Caddy.
+
+example.tld  {
   log /var/log/caddy/pleroma_access.log
   errors /var/log/caddy/pleroma_error.log
 
@@ -9,34 +15,12 @@ social.domain.tld  {
     transparent
   }
 
-  tls user@domain.tld {
+  tls {
     # Remove the rest of the lines in here, if you want to support older devices
     key_type p256
     ciphers ECDHE-ECDSA-WITH-CHACHA20-POLY1305 ECDHE-RSA-WITH-CHACHA20-POLY1305 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256
   }
 
-  header / {
-    X-XSS-Protection "1; mode=block"
-    X-Frame-Options "DENY"
-    X-Content-Type-Options "nosniff"
-    Referrer-Policy "same-origin"
-    Strict-Transport-Security "max-age=31536000; includeSubDomains;"
-    Expect-CT "enforce, max-age=2592000"
-    Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'; img-src 'self' data: https:; media-src 'self' https:; style-src 'self' 'unsafe-inline'; font-src 'self'; script-src 'self'; connect-src 'self' wss://social.domain.tld; upgrade-insecure-requests;"
-  }
-
-  # If you do not want remote frontends to be able to access your Pleroma backend server, remove these lines.
-  # If you want to allow all origins access, remove the origin lines.
-  # To use this directive, you need the http.cors plugin for Caddy.
-  cors / {
-    origin https://halcyon.domain.tld
-    origin https://pinafore.domain.tld
-    methods POST,PUT,DELETE,GET,PATCH,OPTIONS
-    allowed_headers Authorization,Content-Type,Idempotency-Key
-    exposed_headers Link,X-RateLimit-Reset,X-RateLimit-Limit,X-RateLimit-Remaining,X-Request-Id
-  }
-  # Stop removing lines here.
-
   # If you do not want to use the mediaproxy function, remove these lines.
   # To use this directive, you need the http.cache plugin for Caddy.
   cache {

installation/pleroma-apache.conf

@@ -1,24 +1,31 @@
-#Example configuration for when Apache httpd and Pleroma are on the same host.
-#Needed modules: headers proxy proxy_http proxy_wstunnel rewrite ssl
-#This assumes a Debian style Apache config. Put this in /etc/apache2/sites-available
-#Install your TLS certificate, possibly using Let's Encrypt.
-#Replace 'pleroma.example.com' with your instance's domain wherever it appears
+# default Apache site config for Pleroma
+#
+# needed modules: define headers proxy proxy_http proxy_wstunnel rewrite ssl
+#
+# Simple installation instructions:
+# 1. Install your TLS certificate, possibly using Let's Encrypt.
+# 2. Replace 'example.tld' with your instance's domain wherever it appears.
+# 3. This assumes a Debian style Apache config. Copy this file to
+#    /etc/apache2/sites-available/ and then add a symlink to it in
+#    /etc/apache2/sites-enabled/ by running 'a2ensite pleroma-apache.conf', then restart Apache.
 
-ServerName pleroma.example.com
+Define servername example.tld
+
+ServerName ${servername}
 ServerTokens Prod
 
 ErrorLog ${APACHE_LOG_DIR}/error.log
 CustomLog ${APACHE_LOG_DIR}/access.log combined
 
 <VirtualHost *:80>
-    Redirect permanent / https://pleroma.example.com
+    Redirect permanent / https://${servername}
 </VirtualHost>
 
 <VirtualHost *:443>
     SSLEngine on
-    SSLCertificateFile      /etc/letsencrypt/live/pleroma.example.com/cert.pem
-    SSLCertificateKeyFile   /etc/letsencrypt/live/pleroma.example.com/privkey.pem
-    SSLCertificateChainFile /etc/letsencrypt/live/pleroma.example.com/fullchain.pem
+    SSLCertificateFile      /etc/letsencrypt/live/${servername}/cert.pem
+    SSLCertificateKeyFile   /etc/letsencrypt/live/${servername}/privkey.pem
+    SSLCertificateChainFile /etc/letsencrypt/live/${servername}/fullchain.pem
 
     # Mozilla modern configuration, tweak to your needs
     SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
@@ -27,15 +34,6 @@ CustomLog ${APACHE_LOG_DIR}/access.log combined
     SSLCompression          off
     SSLSessionTickets       off
 
-    Header always set X-Xss-Protection "1; mode=block"
-    Header always set X-Frame-Options "DENY"
-    Header always set X-Content-Type-Options "nosniff"
-    Header always set Referrer-Policy same-origin
-    Header always set Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'; img-src 'self' data: https:; media-src 'self' https:; style-src 'self' 'unsafe-inline'; font-src 'self'; script-src 'self'; connect-src 'self' wss://pleroma.example.tld; upgrade-insecure-requests;"
-
-    # Uncomment this only after you get HTTPS working.
-    # Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
-
     RewriteEngine On
     RewriteCond %{HTTP:Connection} Upgrade [NC]
     RewriteCond %{HTTP:Upgrade} websocket [NC]
@@ -45,7 +43,7 @@ CustomLog ${APACHE_LOG_DIR}/access.log combined
     ProxyPass / http://localhost:4000/
     ProxyPassReverse / http://localhost:4000/
 
-    RequestHeader set Host "pleroma.example.com"
+    RequestHeader set Host ${servername}
     ProxyPreserveHost On
 </VirtualHost>
 
@@ -53,4 +51,4 @@ CustomLog ${APACHE_LOG_DIR}/access.log combined
 SSLUseStapling          on
 SSLStaplingResponderTimeout 5
 SSLStaplingReturnResponderErrors off
-SSLStaplingCache        shmcb:/var/run/ocsp(128000)
+SSLStaplingCache        shmcb:/var/run/ocsp(128000)

installation/pleroma.nginx

@@ -10,8 +10,8 @@ proxy_cache_path /tmp/pleroma-media-cache levels=1:2 keys_zone=pleroma_media_cac
                  inactive=720m use_temp_path=off;
 
 server {
-    listen         80;
     server_name    example.tld;
+    listen         80;
     return         301 https://$server_name$request_uri;
 
     # Uncomment this if you need to use the 'webroot' method with certbot. Make sure
@@ -46,7 +46,7 @@ server {
     ssl_ecdh_curve X25519:prime256v1:secp384r1:secp521r1;
     ssl_stapling on;
     ssl_stapling_verify on;
-    
+
     server_name example.tld;
 
     gzip_vary on;
@@ -60,28 +60,6 @@ server {
     client_max_body_size 16m;
 
     location / {
-        # if you do not want remote frontends to be able to access your Pleroma backend
-        # server, remove these lines.
-        add_header 'Access-Control-Allow-Origin' '*' always;
-        add_header 'Access-Control-Allow-Methods' 'POST, PUT, DELETE, GET, PATCH, OPTIONS' always;
-        add_header 'Access-Control-Allow-Headers' 'Authorization, Content-Type, Idempotency-Key' always;
-        add_header 'Access-Control-Expose-Headers' 'Link, X-RateLimit-Reset, X-RateLimit-Limit, X-RateLimit-Remaining, X-Request-Id' always;
-        if ($request_method = OPTIONS) {
-            return 204;
-        }
-        # stop removing lines here.
-
-        add_header X-XSS-Protection "1; mode=block" always;
-        add_header X-Permitted-Cross-Domain-Policies "none" always;
-        add_header X-Frame-Options "DENY" always;
-        add_header X-Content-Type-Options "nosniff" always;
-        add_header Referrer-Policy "same-origin" always;
-        add_header X-Download-Options "noopen" always;
-        add_header Content-Security-Policy "default-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'; img-src 'self' data: https:; media-src 'self' https:; style-src 'self' 'unsafe-inline'; font-src 'self'; script-src 'self'; connect-src 'self' wss://example.tld; upgrade-insecure-requests;" always;
-        
-        # Uncomment this only after you get HTTPS working.
-        # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
-
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";
@@ -92,10 +70,12 @@ server {
         client_max_body_size 16m;
     }
 
-    location /proxy {
+    location ~ ^/(media|proxy) {
         proxy_cache pleroma_media_cache;
         proxy_cache_lock on;
         proxy_ignore_client_abort on;
+        proxy_buffering off;
+        chunked_transfer_encoding on;
         proxy_pass http://localhost:4000;
     }
 }

installation/pleroma.service

@@ -6,10 +6,21 @@ After=network.target postgresql.service
 User=pleroma
 WorkingDirectory=/home/pleroma/pleroma
 Environment="HOME=/home/pleroma"
+Environment="MIX_ENV=prod"
 ExecStart=/usr/local/bin/mix phx.server
 ExecReload=/bin/kill $MAINPID
 KillMode=process
 Restart=on-failure
 
+; Some security directives.
+; Use private /tmp and /var/tmp folders inside a new file system namespace, which are discarded after the process stops.
+PrivateTmp=true
+; Mount /usr, /boot, and /etc as read-only for processes invoked by this service.
+ProtectSystem=full
+; Sets up a new /dev mount for the process and only adds API pseudo devices like /dev/null, /dev/zero or /dev/random but not physical devices. Disabled by default because it may not work on devices like the Raspberry Pi.
+PrivateDevices=false
+; Ensures that the service process and all its children can never gain new privileges through execve().
+NoNewPrivileges=true
+
 [Install]
 WantedBy=multi-user.target

installation/pleroma.vcl

@@ -119,13 +119,3 @@ sub vcl_pipe {
         set bereq.http.connection = req.http.connection;
     }
 }
-
-sub vcl_deliver {
-  set resp.http.X-Frame-Options = "DENY";
-  set resp.http.X-XSS-Protection = "1; mode=block";
-  set resp.http.X-Content-Type-Options = "nosniff";
-  set resp.http.Referrer-Policy = "same-origin";
-  set resp.http.Content-Security-Policy = "default-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'; img-src 'self' data: https:; media-src 'self' https:; style-src 'self' 'unsafe-inline'; font-src 'self'; script-src 'self'; connect-src 'self' wss://" + req.http.host + "; upgrade-insecure-requests;";
-  # Uncomment this only after you get HTTPS working.
-  # set resp.http.Strict-Transport-Security= "max-age=31536000; includeSubDomains";
-}

lib/mix/tasks/deactivate_user.ex

@@ -1,13 +0,0 @@
-defmodule Mix.Tasks.DeactivateUser do
-  use Mix.Task
-  alias Pleroma.User
-
-  @shortdoc "Toggle deactivation status for a user"
-  def run([nickname]) do
-    Mix.Task.run("app.start")
-
-    with user <- User.get_by_nickname(nickname) do
-      User.deactivate(user)
-    end
-  end
-end

lib/mix/tasks/fix_ap_users.ex

@@ -1,28 +0,0 @@
-defmodule Mix.Tasks.FixApUsers do
-  use Mix.Task
-  import Ecto.Query
-  alias Pleroma.{Repo, User}
-
-  @shortdoc "Grab all ap users again"
-  def run([]) do
-    Mix.Task.run("app.start")
-
-    q =
-      from(
-        u in User,
-        where: fragment("? @> ?", u.info, ^%{"ap_enabled" => true}),
-        where: u.local == false
-      )
-
-    users = Repo.all(q)
-
-    Enum.each(users, fn user ->
-      try do
-        IO.puts("Fetching #{user.nickname}")
-        Pleroma.Web.ActivityPub.Transmogrifier.upgrade_user_from_ap_id(user.ap_id, false)
-      rescue
-        e -> IO.inspect(e)
-      end
-    end)
-  end
-end

lib/mix/tasks/generate_config.ex

@@ -1,39 +0,0 @@
-defmodule Mix.Tasks.GenerateConfig do
-  use Mix.Task
-
-  @shortdoc "Generates a new config"
-  def run(_) do
-    IO.puts("Answer a few questions to generate a new config\n")
-    IO.puts("--- THIS WILL OVERWRITE YOUR config/generated_config.exs! ---\n")
-    domain = IO.gets("What is your domain name? (e.g. pleroma.soykaf.com): ") |> String.trim()
-    name = IO.gets("What is the name of your instance? (e.g. Pleroma/Soykaf): ") |> String.trim()
-    email = IO.gets("What's your admin email address: ") |> String.trim()
-
-    secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
-    dbpass = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
-
-    resultSql = EEx.eval_file("lib/mix/tasks/sample_psql.eex", dbpass: dbpass)
-
-    result =
-      EEx.eval_file(
-        "lib/mix/tasks/sample_config.eex",
-        domain: domain,
-        email: email,
-        name: name,
-        secret: secret,
-        dbpass: dbpass
-      )
-
-    IO.puts(
-      "\nWriting config to config/generated_config.exs.\n\nCheck it and configure your database, then copy it to either config/dev.secret.exs or config/prod.secret.exs"
-    )
-
-    File.write("config/generated_config.exs", result)
-
-    IO.puts(
-      "\nWriting setup_db.psql, please run it as postgre superuser, i.e.: sudo su postgres -c 'psql -f config/setup_db.psql'"
-    )
-
-    File.write("config/setup_db.psql", resultSql)
-  end
-end

lib/mix/tasks/generate_invite_token.ex

@@ -1,25 +0,0 @@
-defmodule Mix.Tasks.GenerateInviteToken do
-  use Mix.Task
-
-  @shortdoc "Generate invite token for user"
-  def run([]) do
-    Mix.Task.run("app.start")
-
-    with {:ok, token} <- Pleroma.UserInviteToken.create_token() do
-      IO.puts("Generated user invite token")
-
-      IO.puts(
-        "Url: #{
-          Pleroma.Web.Router.Helpers.redirect_url(
-            Pleroma.Web.Endpoint,
-            :registration_page,
-            token.token
-          )
-        }"
-      )
-    else
-      _ ->
-        IO.puts("Error creating token")
-    end
-  end
-end

lib/mix/tasks/generate_password_reset.ex

@@ -1,27 +0,0 @@
-defmodule Mix.Tasks.GeneratePasswordReset do
-  use Mix.Task
-  alias Pleroma.User
-
-  @shortdoc "Generate password reset link for user"
-  def run([nickname]) do
-    Mix.Task.run("app.start")
-
-    with %User{local: true} = user <- User.get_by_nickname(nickname),
-         {:ok, token} <- Pleroma.PasswordResetToken.create_token(user) do
-      IO.puts("Generated password reset token for #{user.nickname}")
-
-      IO.puts(
-        "Url: #{
-          Pleroma.Web.Router.Helpers.util_url(
-            Pleroma.Web.Endpoint,
-            :show_password_reset,
-            token.token
-          )
-        }"
-      )
-    else
-      _ ->
-        IO.puts("No local user #{nickname}")
-    end
-  end
-end

lib/mix/tasks/make_moderator.ex

@@ -1,30 +0,0 @@
-defmodule Mix.Tasks.SetModerator do
-  use Mix.Task
-  import Mix.Ecto
-  alias Pleroma.{Repo, User}
-
-  @shortdoc "Set moderator status"
-  def run([nickname | rest]) do
-    Application.ensure_all_started(:pleroma)
-
-    moderator =
-      case rest do
-        [moderator] -> moderator == "true"
-        _ -> true
-      end
-
-    with %User{local: true} = user <- User.get_by_nickname(nickname) do
-      info =
-        user.info
-        |> Map.put("is_moderator", !!moderator)
-
-      cng = User.info_changeset(user, %{info: info})
-      {:ok, user} = User.update_and_set_cache(cng)
-
-      IO.puts("Moderator status of #{nickname}: #{user.info["is_moderator"]}")
-    else
-      _ ->
-        IO.puts("No local user #{nickname}")
-    end
-  end
-end

lib/mix/tasks/pleroma/common.ex

@@ -0,0 +1,24 @@
+defmodule Mix.Tasks.Pleroma.Common do
+  @doc "Common functions to be reused in mix tasks"
+  def start_pleroma do
+    Mix.Task.run("app.start")
+  end
+
+  def get_option(options, opt, prompt, defval \\ nil, defname \\ nil) do
+    Keyword.get(options, opt) ||
+      case Mix.shell().prompt("#{prompt} [#{defname || defval}]") do
+        "\n" ->
+          case defval do
+            nil -> get_option(options, opt, prompt, defval)
+            defval -> defval
+          end
+
+        opt ->
+          opt |> String.trim()
+      end
+  end
+
+  def escape_sh_path(path) do
+    ~S(') <> String.replace(path, ~S('), ~S(\')) <> ~S(')
+  end
+end

lib/mix/tasks/pleroma/instance.ex

@@ -0,0 +1,156 @@
+defmodule Mix.Tasks.Pleroma.Instance do
+  use Mix.Task
+  alias Mix.Tasks.Pleroma.Common
+
+  @shortdoc "Manages Pleroma instance"
+  @moduledoc """
+  Manages Pleroma instance.
+
+  ## Generate a new instance config.
+
+    mix pleroma.instance gen [OPTION...]
+
+  If any options are left unspecified, you will be prompted interactively
+
+  ## Options
+
+  - `-f`, `--force` - overwrite any output files
+  - `-o PATH`, `--output PATH` - the output file for the generated configuration
+  - `--output-psql PATH` - the output file for the generated PostgreSQL setup
+  - `--domain DOMAIN` - the domain of your instance
+  - `--instance-name INSTANCE_NAME` - the name of your instance
+  - `--admin-email ADMIN_EMAIL` - the email address of the instance admin
+  - `--dbhost HOSTNAME` - the hostname of the PostgreSQL database to use
+  - `--dbname DBNAME` - the name of the database to use
+  - `--dbuser DBUSER` - the user (aka role) to use for the database connection
+  - `--dbpass DBPASS` - the password to use for the database connection
+  """
+
+  def run(["gen" | rest]) do
+    {options, [], []} =
+      OptionParser.parse(
+        rest,
+        strict: [
+          force: :boolean,
+          output: :string,
+          output_psql: :string,
+          domain: :string,
+          instance_name: :string,
+          admin_email: :string,
+          dbhost: :string,
+          dbname: :string,
+          dbuser: :string,
+          dbpass: :string
+        ],
+        aliases: [
+          o: :output,
+          f: :force
+        ]
+      )
+
+    paths =
+      [config_path, psql_path] = [
+        Keyword.get(options, :output, "config/generated_config.exs"),
+        Keyword.get(options, :output_psql, "config/setup_db.psql")
+      ]
+
+    will_overwrite = Enum.filter(paths, &File.exists?/1)
+    proceed? = Enum.empty?(will_overwrite) or Keyword.get(options, :force, false)
+
+    unless not proceed? do
+      domain =
+        Common.get_option(
+          options,
+          :domain,
+          "What domain will your instance use? (e.g pleroma.soykaf.com)"
+        )
+
+      name =
+        Common.get_option(
+          options,
+          :name,
+          "What is the name of your instance? (e.g. Pleroma/Soykaf)"
+        )
+
+      email = Common.get_option(options, :admin_email, "What is your admin email address?")
+
+      dbhost =
+        Common.get_option(options, :dbhost, "What is the hostname of your database?", "localhost")
+
+      dbname =
+        Common.get_option(options, :dbname, "What is the name of your database?", "pleroma_dev")
+
+      dbuser =
+        Common.get_option(
+          options,
+          :dbuser,
+          "What is the user used to connect to your database?",
+          "pleroma"
+        )
+
+      dbpass =
+        Common.get_option(
+          options,
+          :dbpass,
+          "What is the password used to connect to your database?",
+          :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64),
+          "autogenerated"
+        )
+
+      secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
+      {web_push_public_key, web_push_private_key} = :crypto.generate_key(:ecdh, :prime256v1)
+
+      result_config =
+        EEx.eval_file(
+          "sample_config.eex" |> Path.expand(__DIR__),
+          domain: domain,
+          email: email,
+          name: name,
+          dbhost: dbhost,
+          dbname: dbname,
+          dbuser: dbuser,
+          dbpass: dbpass,
+          version: Pleroma.Mixfile.project() |> Keyword.get(:version),
+          secret: secret,
+          web_push_public_key: Base.url_encode64(web_push_public_key, padding: false),
+          web_push_private_key: Base.url_encode64(web_push_private_key, padding: false)
+        )
+
+      result_psql =
+        EEx.eval_file(
+          "sample_psql.eex" |> Path.expand(__DIR__),
+          dbname: dbname,
+          dbuser: dbuser,
+          dbpass: dbpass
+        )
+
+      Mix.shell().info(
+        "Writing config to #{config_path}. You should rename it to config/prod.secret.exs or config/dev.secret.exs."
+      )
+
+      File.write(config_path, result_config)
+      Mix.shell().info("Writing #{psql_path}.")
+      File.write(psql_path, result_psql)
+
+      Mix.shell().info(
+        "\n" <>
+          """
+          To get started:
+          1. Verify the contents of the generated files.
+          2. Run `sudo -u postgres psql -f #{Common.escape_sh_path(psql_path)}`.
+          """ <>
+          if config_path in ["config/dev.secret.exs", "config/prod.secret.exs"] do
+            ""
+          else
+            "3. Run `mv #{Common.escape_sh_path(config_path)} 'config/prod.secret.exs'`."
+          end
+      )
+    else
+      Mix.shell().error(
+        "The task would have overwritten the following files:\n" <>
+          (Enum.map(paths, &"- #{&1}\n") |> Enum.join("")) <>
+          "Rerun with `--force` to overwrite them."
+      )
+    end
+  end
+end

lib/mix/tasks/pleroma/relay.ex

@@ -0,0 +1,43 @@
+defmodule Mix.Tasks.Pleroma.Relay do
+  use Mix.Task
+  alias Pleroma.Web.ActivityPub.Relay
+  alias Mix.Tasks.Pleroma.Common
+
+  @shortdoc "Manages remote relays"
+  @moduledoc """
+  Manages remote relays
+
+  ## Follow a remote relay
+
+  ``mix pleroma.relay follow <relay_url>``
+
+  Example: ``mix pleroma.relay follow  https://example.org/relay``
+
+  ## Unfollow a remote relay
+
+  ``mix pleroma.relay unfollow <relay_url>``
+
+  Example: ``mix pleroma.relay unfollow https://example.org/relay``
+  """
+  def run(["follow", target]) do
+    Common.start_pleroma()
+
+    with {:ok, _activity} <- Relay.follow(target) do
+      # put this task to sleep to allow the genserver to push out the messages
+      :timer.sleep(500)
+    else
+      {:error, e} -> Mix.shell().error("Error while following #{target}: #{inspect(e)}")
+    end
+  end
+
+  def run(["unfollow", target]) do
+    Common.start_pleroma()
+
+    with {:ok, _activity} <- Relay.unfollow(target) do
+      # put this task to sleep to allow the genserver to push out the messages
+      :timer.sleep(500)
+    else
+      {:error, e} -> Mix.shell().error("Error while following #{target}: #{inspect(e)}")
+    end
+  end
+end

lib/mix/tasks/pleroma/sample_config.eex

@@ -1,3 +1,8 @@
+# Pleroma instance configuration
+
+# NOTE: This file should not be committed to a repo or otherwise made public
+# without removing sensitive information.
+
 use Mix.Config
 
 config :pleroma, Pleroma.Web.Endpoint,
@@ -16,15 +21,24 @@ config :pleroma, :media_proxy,
   redirect_on_failure: true
   #base_url: "https://cache.pleroma.social"
 
-# Configure your database
 config :pleroma, Pleroma.Repo,
   adapter: Ecto.Adapters.Postgres,
-  username: "pleroma",
+  username: "<%= dbuser %>",
   password: "<%= dbpass %>",
-  database: "pleroma_dev",
-  hostname: "localhost",
+  database: "<%= dbname %>",
+  hostname: "<%= dbhost %>",
   pool_size: 10
 
+# Configure web push notifications
+config :web_push_encryption, :vapid_details,
+  subject: "mailto:<%= email %>",
+  public_key: "<%= web_push_public_key %>",
+  private_key: "<%= web_push_private_key %>"
+
+# Enable Strict-Transport-Security once SSL is working:
+# config :pleroma, :http_security,
+#   sts: true
+
 # Configure S3 support if desired.
 # The public S3 endpoint is different depending on region and provider,
 # consult your S3 provider's documentation for details on what to use.
@@ -46,9 +60,9 @@ config :pleroma, Pleroma.Repo,
 
 
 # Configure Openstack Swift support if desired.
-# 
-# Many openstack deployments are different, so config is left very open with 
-# no assumptions made on which provider you're using. This should allow very 
+#
+# Many openstack deployments are different, so config is left very open with
+# no assumptions made on which provider you're using. This should allow very
 # wide support without needing separate handlers for OVH, Rackspace, etc.
 #
 # config :pleroma, Pleroma.Uploaders.Swift,

lib/mix/tasks/pleroma/sample_psql.eex

@@ -0,0 +1,7 @@
+CREATE USER pleroma WITH ENCRYPTED PASSWORD '<%= dbpass %>';
+CREATE DATABASE pleroma_dev OWNER pleroma;
+\c pleroma_dev;
+--Extensions made by ecto.migrate that need superuser access
+CREATE EXTENSION IF NOT EXISTS citext;
+CREATE EXTENSION IF NOT EXISTS pg_trgm;
+CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

lib/mix/tasks/pleroma/uploads.ex

@@ -0,0 +1,103 @@
+defmodule Mix.Tasks.Pleroma.Uploads do
+  use Mix.Task
+  alias Pleroma.{Upload, Uploaders.Local}
+  alias Mix.Tasks.Pleroma.Common
+  require Logger
+
+  @log_every 50
+
+  @shortdoc "Migrates uploads from local to remote storage"
+  @moduledoc """
+   Manages uploads
+
+   ## Migrate uploads from local to remote storage
+       mix pleroma.uploads migrate_local TARGET_UPLOADER [OPTIONS...]
+   Options:
+   - `--delete` - delete local uploads after migrating them to the target uploader
+
+
+   A list of avalible uploaders can be seen in config.exs
+  """
+  def run(["migrate_local", target_uploader | args]) do
+    delete? = Enum.member?(args, "--delete")
+    Common.start_pleroma()
+    local_path = Pleroma.Config.get!([Local, :uploads])
+    uploader = Module.concat(Pleroma.Uploaders, target_uploader)
+
+    unless Code.ensure_loaded?(uploader) do
+      raise("The uploader #{inspect(uploader)} is not an existing/loaded module.")
+    end
+
+    target_enabled? = Pleroma.Config.get([Upload, :uploader]) == uploader
+
+    unless target_enabled? do
+      Pleroma.Config.put([Upload, :uploader], uploader)
+    end
+
+    Mix.shell().info("Migrating files from local #{local_path} to #{to_string(uploader)}")
+
+    if delete? do
+      Mix.shell().info(
+        "Attention: uploaded files will be deleted, hope you have backups! (--delete ; cancel with ^C)"
+      )
+
+      :timer.sleep(:timer.seconds(5))
+    end
+
+    uploads =
+      File.ls!(local_path)
+      |> Enum.map(fn id ->
+        root_path = Path.join(local_path, id)
+
+        cond do
+          File.dir?(root_path) ->
+            files = for file <- File.ls!(root_path), do: {id, file, Path.join([root_path, file])}
+
+            case List.first(files) do
+              {id, file, path} ->
+                {%Pleroma.Upload{id: id, name: file, path: id <> "/" <> file, tempfile: path},
+                 root_path}
+
+              _ ->
+                nil
+            end
+
+          File.exists?(root_path) ->
+            file = Path.basename(id)
+            hash = Path.rootname(id)
+            {%Pleroma.Upload{id: hash, name: file, path: file, tempfile: root_path}, root_path}
+
+          true ->
+            nil
+        end
+      end)
+      |> Enum.filter(& &1)
+
+    total_count = length(uploads)
+    Mix.shell().info("Found #{total_count} uploads")
+
+    uploads
+    |> Task.async_stream(
+      fn {upload, root_path} ->
+        case Upload.store(upload, uploader: uploader, filters: [], size_limit: nil) do
+          {:ok, _} ->
+            if delete?, do: File.rm_rf!(root_path)
+            Logger.debug("uploaded: #{inspect(upload.path)} #{inspect(upload)}")
+            :ok
+
+          error ->
+            Mix.shell().error("failed to upload #{inspect(upload.path)}: #{inspect(error)}")
+        end
+      end,
+      timeout: 150_000
+    )
+    |> Stream.chunk_every(@log_every)
+    |> Enum.reduce(0, fn done, count ->
+      count = count + length(done)
+      Mix.shell().info("Uploaded #{count}/#{total_count} files")
+      count
+    end)
+
+    Mix.shell().info("Done!")
+  end
+end

lib/mix/tasks/pleroma/user.ex

@@ -0,0 +1,293 @@
+defmodule Mix.Tasks.Pleroma.User do
+  use Mix.Task
+  import Ecto.Changeset
+  alias Pleroma.{Repo, User}
+  alias Mix.Tasks.Pleroma.Common
+
+  @shortdoc "Manages Pleroma users"
+  @moduledoc """
+  Manages Pleroma users.
+
+  ## Create a new user.
+
+      mix pleroma.user new NICKNAME EMAIL [OPTION...]
+
+  Options:
+  - `--name NAME` - the user's name (i.e., "Lain Iwakura")
+  - `--bio BIO` - the user's bio
+  - `--password PASSWORD` - the user's password
+  - `--moderator`/`--no-moderator` - whether the user is a moderator
+  - `--admin`/`--no-admin` - whether the user is an admin
+
+  ## Generate an invite link.
+
+      mix pleroma.user invite
+
+  ## Delete the user's account.
+
+      mix pleroma.user rm NICKNAME
+
+  ## Deactivate or activate the user's account.
+
+      mix pleroma.user toggle_activated NICKNAME
+
+  ## Unsubscribe local users from user's account and deactivate it
+
+      mix pleroma.user unsubscribe NICKNAME
+
+  ## Create a password reset link.
+
+      mix pleroma.user reset_password NICKNAME
+
+  ## Set the value of the given user's settings.
+
+      mix pleroma.user set NICKNAME [OPTION...]
+
+  Options:
+  - `--locked`/`--no-locked` - whether the user's account is locked
+  - `--moderator`/`--no-moderator` - whether the user is a moderator
+  - `--admin`/`--no-admin` - whether the user is an admin
+  """
+  def run(["new", nickname, email | rest]) do
+    {options, [], []} =
+      OptionParser.parse(
+        rest,
+        strict: [
+          name: :string,
+          bio: :string,
+          password: :string,
+          moderator: :boolean,
+          admin: :boolean
+        ]
+      )
+
+    name = Keyword.get(options, :name, nickname)
+    bio = Keyword.get(options, :bio, "")
+
+    {password, generated_password?} =
+      case Keyword.get(options, :password) do
+        nil ->
+          {:crypto.strong_rand_bytes(16) |> Base.encode64(), true}
+
+        password ->
+          {password, false}
+      end
+
+    moderator? = Keyword.get(options, :moderator, false)
+    admin? = Keyword.get(options, :admin, false)
+
+    Mix.shell().info("""
+    A user will be created with the following information:
+      - nickname: #{nickname}
+      - email: #{email}
+      - password: #{
+      if(generated_password?, do: "[generated; a reset link will be created]", else: password)
+    }
+      - name: #{name}
+      - bio: #{bio}
+      - moderator: #{if(moderator?, do: "true", else: "false")}
+      - admin: #{if(admin?, do: "true", else: "false")}
+    """)
+
+    proceed? = Mix.shell().yes?("Continue?")
+
+    unless not proceed? do
+      Common.start_pleroma()
+
+      params =
+        %{
+          nickname: nickname,
+          email: email,
+          password: password,
+          password_confirmation: password,
+          name: name,
+          bio: bio
+        }
+        |> IO.inspect()
+
+      user = User.register_changeset(%User{}, params)
+      Repo.insert!(user)
+
+      Mix.shell().info("User #{nickname} created")
+
+      if moderator? do
+        run(["set", nickname, "--moderator"])
+      end
+
+      if admin? do
+        run(["set", nickname, "--admin"])
+      end
+
+      if generated_password? do
+        run(["reset_password", nickname])
+      end
+    else
+      Mix.shell().info("User will not be created.")
+    end
+  end
+
+  def run(["rm", nickname]) do
+    Common.start_pleroma()
+
+    with %User{local: true} = user <- User.get_by_nickname(nickname) do
+      User.delete(user)
+      Mix.shell().info("User #{nickname} deleted.")
+    else
+      _ ->
+        Mix.shell().error("No local user #{nickname}")
+    end
+  end
+
+  def run(["toggle_activated", nickname]) do
+    Common.start_pleroma()
+
+    with %User{} = user <- User.get_by_nickname(nickname) do
+      User.deactivate(user, !user.info["deactivated"])
+      Mix.shell().info("Activation status of #{nickname}: #{user.info["deactivated"]}")
+    else
+      _ ->
+        Mix.shell().error("No user #{nickname}")
+    end
+  end
+
+  def run(["reset_password", nickname]) do
+    Common.start_pleroma()
+
+    with %User{local: true} = user <- User.get_by_nickname(nickname),
+         {:ok, token} <- Pleroma.PasswordResetToken.create_token(user) do
+      Mix.shell().info("Generated password reset token for #{user.nickname}")
+
+      IO.puts(
+        "URL: #{
+          Pleroma.Web.Router.Helpers.util_url(
+            Pleroma.Web.Endpoint,
+            :show_password_reset,
+            token.token
+          )
+        }"
+      )
+    else
+      _ ->
+        Mix.shell().error("No local user #{nickname}")
+    end
+  end
+
+  def run(["unsubscribe", nickname]) do
+    Common.start_pleroma()
+
+    with %User{} = user <- User.get_by_nickname(nickname) do
+      Mix.shell().info("Deactivating #{user.nickname}")
+      User.deactivate(user)
+
+      {:ok, friends} = User.get_friends(user)
+
+      Enum.each(friends, fn friend ->
+        user = Repo.get(User, user.id)
+
+        Mix.shell().info("Unsubscribing #{friend.nickname} from #{user.nickname}")
+        User.unfollow(user, friend)
+      end)
+
+      :timer.sleep(500)
+
+      user = Repo.get(User, user.id)
+
+      if length(user.following) == 0 do
+        Mix.shell().info("Successfully unsubscribed all followers from #{user.nickname}")
+      end
+    else
+      _ ->
+        Mix.shell().error("No user #{nickname}")
+    end
+  end
+
+  def run(["set", nickname | rest]) do
+    Common.start_pleroma()
+
+    {options, [], []} =
+      OptionParser.parse(
+        rest,
+        strict: [
+          moderator: :boolean,
+          admin: :boolean,
+          locked: :boolean
+        ]
+      )
+
+    with %User{local: true} = user <- User.get_by_nickname(nickname) do
+      case Keyword.get(options, :moderator) do
+        nil -> nil
+        value -> set_moderator(user, value)
+      end
+
+      case Keyword.get(options, :locked) do
+        nil -> nil
+        value -> set_locked(user, value)
+      end
+
+      case Keyword.get(options, :admin) do
+        nil -> nil
+        value -> set_admin(user, value)
+      end
+    else
+      _ ->
+        Mix.shell().error("No local user #{nickname}")
+    end
+  end
+
+  def run(["invite"]) do
+    Common.start_pleroma()
+
+    with {:ok, token} <- Pleroma.UserInviteToken.create_token() do
+      Mix.shell().info("Generated user invite token")
+
+      url =
+        Pleroma.Web.Router.Helpers.redirect_url(
+          Pleroma.Web.Endpoint,
+          :registration_page,
+          token.token
+        )
+
+      IO.puts(url)
+    else
+      _ ->
+        Mix.shell().error("Could not create invite token.")
+    end
+  end
+
+  defp set_moderator(user, value) do
+    info_cng = User.Info.admin_api_update(user.info, %{is_moderator: value})
+
+    user_cng =
+      Ecto.Changeset.change(user)
+      |> put_embed(:info, info_cng)
+
+    {:ok, user} = User.update_and_set_cache(user_cng)
+
+    Mix.shell().info("Moderator status of #{user.nickname}: #{user.info.is_moderator}")
+  end
+
+  defp set_admin(user, value) do
+    info_cng = User.Info.admin_api_update(user.info, %{is_admin: value})
+
+    user_cng =
+      Ecto.Changeset.change(user)
+      |> put_embed(:info, info_cng)
+
+    {:ok, user} = User.update_and_set_cache(user_cng)
+
+    Mix.shell().info("Admin status of #{user.nickname}: #{user.info.is_moderator}")
+  end
+
+  defp set_locked(user, value) do
+    info_cng = User.Info.user_upgrade(user.info, %{locked: value})
+
+    user_cng =
+      Ecto.Changeset.change(user)
+      |> put_embed(:info, info_cng)
+
+    {:ok, user} = User.update_and_set_cache(user_cng)
+
+    Mix.shell().info("Locked status of #{user.nickname}: #{user.info.locked}")
+  end
+end

lib/mix/tasks/register_user.ex

@@ -1,22 +0,0 @@
-defmodule Mix.Tasks.RegisterUser do
-  use Mix.Task
-  alias Pleroma.{Repo, User}
-
-  @shortdoc "Register user"
-  def run([name, nickname, email, bio, password]) do
-    Mix.Task.run("app.start")
-
-    params = %{
-      name: name,
-      nickname: nickname,
-      email: email,
-      password: password,
-      password_confirmation: password,
-      bio: bio
-    }
-
-    user = User.register_changeset(%User{}, params)
-
-    Repo.insert!(user)
-  end
-end

lib/mix/tasks/relay_follow.ex

@@ -1,15 +0,0 @@
-defmodule Mix.Tasks.RelayFollow do
-  use Mix.Task
-  require Logger
-  alias Pleroma.Web.ActivityPub.Relay
-
-  @shortdoc "Follows a remote relay"
-  def run([target]) do
-    Mix.Task.run("app.start")
-
-    :ok = Relay.follow(target)
-
-    # put this task to sleep to allow the genserver to push out the messages
-    :timer.sleep(500)
-  end
-end

lib/mix/tasks/relay_unfollow.ex

@@ -1,15 +0,0 @@
-defmodule Mix.Tasks.RelayUnfollow do
-  use Mix.Task
-  require Logger
-  alias Pleroma.Web.ActivityPub.Relay
-
-  @shortdoc "Follows a remote relay"
-  def run([target]) do
-    Mix.Task.run("app.start")
-
-    :ok = Relay.unfollow(target)
-
-    # put this task to sleep to allow the genserver to push out the messages
-    :timer.sleep(500)
-  end
-end

lib/mix/tasks/rm_user.ex

@@ -1,13 +0,0 @@
-defmodule Mix.Tasks.RmUser do
-  use Mix.Task
-  alias Pleroma.User
-
-  @shortdoc "Permanently delete a user"
-  def run([nickname]) do
-    Mix.Task.run("app.start")
-
-    with %User{local: true} = user <- User.get_by_nickname(nickname) do
-      User.delete(user)
-    end
-  end
-end

lib/mix/tasks/sample_psql.eex

@@ -1,9 +0,0 @@
-CREATE USER pleroma WITH ENCRYPTED PASSWORD '<%= dbpass %>' CREATEDB;
--- in case someone runs this second time accidentally
-ALTER USER pleroma WITH ENCRYPTED PASSWORD '<%= dbpass %>' CREATEDB;
-CREATE DATABASE pleroma_dev;
-ALTER DATABASE pleroma_dev OWNER TO pleroma;
-\c pleroma_dev;
---Extensions made by ecto.migrate that need superuser access
-CREATE EXTENSION IF NOT EXISTS citext;
-CREATE EXTENSION IF NOT EXISTS pg_trgm;

lib/mix/tasks/set_locked.ex

@@ -1,30 +0,0 @@
-defmodule Mix.Tasks.SetLocked do
-  use Mix.Task
-  import Mix.Ecto
-  alias Pleroma.{Repo, User}
-
-  @shortdoc "Set locked status"
-  def run([nickname | rest]) do
-    ensure_started(Repo, [])
-
-    locked =
-      case rest do
-        [locked] -> locked == "true"
-        _ -> true
-      end
-
-    with %User{local: true} = user <- User.get_by_nickname(nickname) do
-      info =
-        user.info
-        |> Map.put("locked", !!locked)
-
-      cng = User.info_changeset(user, %{info: info})
-      user = Repo.update!(cng)
-
-      IO.puts("locked status of #{nickname}: #{user.info["locked"]}")
-    else
-      _ ->
-        IO.puts("No local user #{nickname}")
-    end
-  end
-end

lib/pleroma/activity.ex

@@ -82,4 +82,10 @@ defmodule Pleroma.Activity do
   def normalize(obj) when is_map(obj), do: Activity.get_by_ap_id(obj["id"])
   def normalize(ap_id) when is_binary(ap_id), do: Activity.get_by_ap_id(ap_id)
   def normalize(_), do: nil
+
+  def get_in_reply_to_activity(%Activity{data: %{"object" => %{"inReplyTo" => ap_id}}}) do
+    get_create_activity_by_object_ap_id(ap_id)
+  end
+
+  def get_in_reply_to_activity(_), do: nil
 end

lib/pleroma/application.ex

@@ -1,10 +1,21 @@
 defmodule Pleroma.Application do
   use Application
+  import Supervisor.Spec
+
+  @name "Pleroma"
+  @version Mix.Project.config()[:version]
+  def name, do: @name
+  def version, do: @version
+  def named_version(), do: @name <> " " <> @version
+
+  def user_agent() do
+    info = "#{Pleroma.Web.base_url()} <#{Pleroma.Config.get([:instance, :email], "")}>"
+    named_version() <> "; " <> info
+  end
 
   # See http://elixir-lang.org/docs/stable/elixir/Application.html
   # for more information on OTP Applications
   def start(_type, _args) do
-    import Supervisor.Spec
     import Cachex.Spec
 
     # Define workers and child supervisors to be supervised
@@ -12,18 +23,31 @@ defmodule Pleroma.Application do
       [
         # Start the Ecto repository
         supervisor(Pleroma.Repo, []),
-        # Start the endpoint when the application starts
-        supervisor(Pleroma.Web.Endpoint, []),
-        # Start your own worker by calling: Pleroma.Worker.start_link(arg1, arg2, arg3)
-        # worker(Pleroma.Worker, [arg1, arg2, arg3]),
-        worker(Cachex, [
-          :user_cache,
+        worker(Pleroma.Emoji, []),
+        worker(
+          Cachex,
           [
-            default_ttl: 25000,
-            ttl_interval: 1000,
-            limit: 2500
-          ]
-        ]),
+            :user_cache,
+            [
+              default_ttl: 25000,
+              ttl_interval: 1000,
+              limit: 2500
+            ]
+          ],
+          id: :cachex_user
+        ),
+        worker(
+          Cachex,
+          [
+            :object_cache,
+            [
+              default_ttl: 25000,
+              ttl_interval: 1000,
+              limit: 2500
+            ]
+          ],
+          id: :cachex_object
+        ),
         worker(
           Cachex,
           [
@@ -39,19 +63,18 @@ defmodule Pleroma.Application do
           ],
           id: :cachex_idem
         ),
+        worker(Pleroma.Web.Federator.RetryQueue, []),
         worker(Pleroma.Web.Federator, []),
-        worker(Pleroma.Gopher.Server, []),
-        worker(Pleroma.Stats, [])
+        worker(Pleroma.Stats, []),
+        worker(Pleroma.Web.Push, [])
       ] ++
-        if Mix.env() == :test,
-          do: [],
-          else:
-            [worker(Pleroma.Web.Streamer, [])] ++
-              if(
-                !chat_enabled(),
-                do: [],
-                else: [worker(Pleroma.Web.ChatChannel.ChatChannelState, [])]
-              )
+        streamer_child() ++
+        chat_child() ++
+        [
+          # Start the endpoint when the application starts
+          supervisor(Pleroma.Web.Endpoint, []),
+          worker(Pleroma.Gopher.Server, [])
+        ]
 
     # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
     # for other strategies and supported options
@@ -59,7 +82,20 @@ defmodule Pleroma.Application do
     Supervisor.start_link(children, opts)
   end
 
-  defp chat_enabled do
-    Application.get_env(:pleroma, :chat, []) |> Keyword.get(:enabled)
+  if Mix.env() == :test do
+    defp streamer_child(), do: []
+    defp chat_child(), do: []
+  else
+    defp streamer_child() do
+      [worker(Pleroma.Web.Streamer, [])]
+    end
+
+    defp chat_child() do
+      if Pleroma.Config.get([:chat, :enabled]) do
+        [worker(Pleroma.Web.ChatChannel.ChatChannelState, [])]
+      else
+        []
+      end
+    end
   end
 end

lib/pleroma/config.ex

@@ -0,0 +1,56 @@
+defmodule Pleroma.Config do
+  defmodule Error do
+    defexception [:message]
+  end
+
+  def get(key), do: get(key, nil)
+
+  def get([key], default), do: get(key, default)
+
+  def get([parent_key | keys], default) do
+    Application.get_env(:pleroma, parent_key)
+    |> get_in(keys) || default
+  end
+
+  def get(key, default) do
+    Application.get_env(:pleroma, key, default)
+  end
+
+  def get!(key) do
+    value = get(key, nil)
+
+    if value == nil do
+      raise(Error, message: "Missing configuration value: #{inspect(key)}")
+    else
+      value
+    end
+  end
+
+  def put([key], value), do: put(key, value)
+
+  def put([parent_key | keys], value) do
+    parent =
+      Application.get_env(:pleroma, parent_key)
+      |> put_in(keys, value)
+
+    Application.put_env(:pleroma, parent_key, parent)
+  end
+
+  def put(key, value) do
+    Application.put_env(:pleroma, key, value)
+  end
+
+  def delete([key]), do: delete(key)
+
+  def delete([parent_key | keys]) do
+    {_, parent} =
+      Application.get_env(:pleroma, parent_key)
+      |> get_and_update_in(keys, fn _ -> :pop end)
+
+    Application.put_env(:pleroma, parent_key, parent)
+  end
+
+  def delete(key) do
+    Application.delete_env(:pleroma, key)
+  end
+end

lib/pleroma/emoji.ex

@@ -0,0 +1,194 @@
+defmodule Pleroma.Emoji do
+  @moduledoc """
+  The emojis are loaded from:
+
+    * the built-in Finmojis (if enabled in configuration),
+    * the files: `config/emoji.txt` and `config/custom_emoji.txt`
+    * glob paths
+
+  This GenServer stores in an ETS table the list of the loaded emojis, and also allows to reload the list at runtime.
+  """
+  use GenServer
+  @ets __MODULE__.Ets
+  @ets_options [:ordered_set, :protected, :named_table, {:read_concurrency, true}]
+
+  @doc false
+  def start_link() do
+    GenServer.start_link(__MODULE__, [], name: __MODULE__)
+  end
+
+  @doc "Reloads the emojis from disk."
+  @spec reload() :: :ok
+  def reload() do
+    GenServer.call(__MODULE__, :reload)
+  end
+
+  @doc "Returns the path of the emoji `name`."
+  @spec get(String.t()) :: String.t() | nil
+  def get(name) do
+    case :ets.lookup(@ets, name) do
+      [{_, path}] -> path
+      _ -> nil
+    end
+  end
+
+  @doc "Returns all the emojos!!"
+  @spec get_all() :: [{String.t(), String.t()}, ...]
+  def get_all() do
+    :ets.tab2list(@ets)
+  end
+
+  @doc false
+  def init(_) do
+    @ets = :ets.new(@ets, @ets_options)
+    GenServer.cast(self(), :reload)
+    {:ok, nil}
+  end
+
+  @doc false
+  def handle_cast(:reload, state) do
+    load()
+    {:noreply, state}
+  end
+
+  @doc false
+  def handle_call(:reload, _from, state) do
+    load()
+    {:reply, :ok, state}
+  end
+
+  @doc false
+  def terminate(_, _) do
+    :ok
+  end
+
+  @doc false
+  def code_change(_old_vsn, state, _extra) do
+    load()
+    {:ok, state}
+  end
+
+  defp load() do
+    emojis =
+      (load_finmoji(Keyword.get(Application.get_env(:pleroma, :instance), :finmoji_enabled)) ++
+         load_from_file("config/emoji.txt") ++
+         load_from_file("config/custom_emoji.txt") ++
+         load_from_globs(
+           Keyword.get(Application.get_env(:pleroma, :emoji, []), :shortcode_globs, [])
+         ))
+      |> Enum.reject(fn value -> value == nil end)
+
+    true = :ets.insert(@ets, emojis)
+    :ok
+  end
+
+  @finmoji [
+    "a_trusted_friend",
+    "alandislands",
+    "association",
+    "auroraborealis",
+    "baby_in_a_box",
+    "bear",
+    "black_gold",
+    "christmasparty",
+    "crosscountryskiing",
+    "cupofcoffee",
+    "education",
+    "fashionista_finns",
+    "finnishlove",
+    "flag",
+    "forest",
+    "four_seasons_of_bbq",
+    "girlpower",
+    "handshake",
+    "happiness",
+    "headbanger",
+    "icebreaker",
+    "iceman",
+    "joulutorttu",
+    "kaamos",
+    "kalsarikannit_f",
+    "kalsarikannit_m",
+    "karjalanpiirakka",
+    "kicksled",
+    "kokko",
+    "lavatanssit",
+    "losthopes_f",
+    "losthopes_m",
+    "mattinykanen",
+    "meanwhileinfinland",
+    "moominmamma",
+    "nordicfamily",
+    "out_of_office",
+    "peacemaker",
+    "perkele",
+    "pesapallo",
+    "polarbear",
+    "pusa_hispida_saimensis",
+    "reindeer",
+    "sami",
+    "sauna_f",
+    "sauna_m",
+    "sauna_whisk",
+    "sisu",
+    "stuck",
+    "suomimainittu",
+    "superfood",
+    "swan",
+    "the_cap",
+    "the_conductor",
+    "the_king",
+    "the_voice",
+    "theoriginalsanta",
+    "tomoffinland",
+    "torillatavataan",
+    "unbreakable",
+    "waiting",
+    "white_nights",
+    "woollysocks"
+  ]
+  defp load_finmoji(true) do
+    Enum.map(@finmoji, fn finmoji ->
+      {finmoji, "/finmoji/128px/#{finmoji}-128.png"}
+    end)
+  end
+
+  defp load_finmoji(_), do: []
+
+  defp load_from_file(file) do
+    if File.exists?(file) do
+      load_from_file_stream(File.stream!(file))
+    else
+      []
+    end
+  end
+
+  defp load_from_file_stream(stream) do
+    stream
+    |> Stream.map(&String.trim/1)
+    |> Stream.map(fn line ->
+      case String.split(line, ~r/,\s*/) do
+        [name, file] -> {name, file}
+        _ -> nil
+      end
+    end)
+    |> Enum.to_list()
+  end
+
+  defp load_from_globs(globs) do
+    static_path = Path.join(:code.priv_dir(:pleroma), "static")
+
+    paths =
+      Enum.map(globs, fn glob ->
+        Path.join(static_path, glob)
+        |> Path.wildcard()
+      end)
+      |> Enum.concat()
+
+    Enum.map(paths, fn path ->
+      shortcode = Path.basename(path, Path.extname(path))
+      external_path = Path.join("/", Path.relative_to(path, static_path))
+      {shortcode, external_path}
+    end)
+  end
+end

lib/pleroma/filter.ex

@@ -1,10 +1,10 @@
 defmodule Pleroma.Filter do
   use Ecto.Schema
   import Ecto.{Changeset, Query}
-  alias Pleroma.{User, Repo, Activity}
+  alias Pleroma.{User, Repo}
 
   schema "filters" do
-    belongs_to(:user, Pleroma.User)
+    belongs_to(:user, User)
     field(:filter_id, :integer)
     field(:hide, :boolean, default: false)
     field(:whole_word, :boolean, default: true)
@@ -26,7 +26,7 @@ defmodule Pleroma.Filter do
     Repo.one(query)
   end
 
-  def get_filters(%Pleroma.User{id: user_id} = user) do
+  def get_filters(%User{id: user_id} = _user) do
     query =
       from(
         f in Pleroma.Filter,
@@ -36,6 +36,34 @@ defmodule Pleroma.Filter do
     Repo.all(query)
   end
 
+  def create(%Pleroma.Filter{user_id: user_id, filter_id: nil} = filter) do
+    # If filter_id wasn't given, use the max filter_id for this user plus 1.
+    # XXX This could result in a race condition if a user tries to add two
+    # different filters for their account from two different clients at the
+    # same time, but that should be unlikely.
+
+    max_id_query =
+      from(
+        f in Pleroma.Filter,
+        where: f.user_id == ^user_id,
+        select: max(f.filter_id)
+      )
+
+    filter_id =
+      case Repo.one(max_id_query) do
+        # Start allocating from 1
+        nil ->
+          1
+
+        max_id ->
+          max_id + 1
+      end
+
+    filter
+    |> Map.put(:filter_id, filter_id)
+    |> Repo.insert()
+  end
+
   def create(%Pleroma.Filter{} = filter) do
     Repo.insert(filter)
   end

lib/pleroma/formatter.ex

@@ -2,6 +2,7 @@ defmodule Pleroma.Formatter do
   alias Pleroma.User
   alias Pleroma.Web.MediaProxy
   alias Pleroma.HTML
+  alias Pleroma.Emoji
 
   @tag_regex ~r/\#\w+/u
   def parse_tags(text, data \\ %{}) do
@@ -28,119 +29,10 @@ defmodule Pleroma.Formatter do
     |> Enum.filter(fn {_match, user} -> user end)
   end
 
-  @finmoji [
-    "a_trusted_friend",
-    "alandislands",
-    "association",
-    "auroraborealis",
-    "baby_in_a_box",
-    "bear",
-    "black_gold",
-    "christmasparty",
-    "crosscountryskiing",
-    "cupofcoffee",
-    "education",
-    "fashionista_finns",
-    "finnishlove",
-    "flag",
-    "forest",
-    "four_seasons_of_bbq",
-    "girlpower",
-    "handshake",
-    "happiness",
-    "headbanger",
-    "icebreaker",
-    "iceman",
-    "joulutorttu",
-    "kaamos",
-    "kalsarikannit_f",
-    "kalsarikannit_m",
-    "karjalanpiirakka",
-    "kicksled",
-    "kokko",
-    "lavatanssit",
-    "losthopes_f",
-    "losthopes_m",
-    "mattinykanen",
-    "meanwhileinfinland",
-    "moominmamma",
-    "nordicfamily",
-    "out_of_office",
-    "peacemaker",
-    "perkele",
-    "pesapallo",
-    "polarbear",
-    "pusa_hispida_saimensis",
-    "reindeer",
-    "sami",
-    "sauna_f",
-    "sauna_m",
-    "sauna_whisk",
-    "sisu",
-    "stuck",
-    "suomimainittu",
-    "superfood",
-    "swan",
-    "the_cap",
-    "the_conductor",
-    "the_king",
-    "the_voice",
-    "theoriginalsanta",
-    "tomoffinland",
-    "torillatavataan",
-    "unbreakable",
-    "waiting",
-    "white_nights",
-    "woollysocks"
-  ]
+  def emojify(text) do
+    emojify(text, Emoji.get_all())
+  end
 
-  @finmoji_with_filenames Enum.map(@finmoji, fn finmoji ->
-                            {finmoji, "/finmoji/128px/#{finmoji}-128.png"}
-                          end)
-
-  @emoji_from_file (with {:ok, default} <- File.read("config/emoji.txt") do
-                      custom =
-                        with {:ok, custom} <- File.read("config/custom_emoji.txt") do
-                          custom
-                        else
-                          _e -> ""
-                        end
-
-                      (default <> "\n" <> custom)
-                      |> String.trim()
-                      |> String.split(~r/\n+/)
-                      |> Enum.map(fn line ->
-                        [name, file] = String.split(line, ~r/,\s*/)
-                        {name, file}
-                      end)
-                    else
-                      _ -> []
-                    end)
-
-  @emoji_from_globs (
-                      static_path = Path.join(:code.priv_dir(:pleroma), "static")
-
-                      globs =
-                        Application.get_env(:pleroma, :emoji, [])
-                        |> Keyword.get(:shortcode_globs, [])
-
-                      paths =
-                        Enum.map(globs, fn glob ->
-                          Path.join(static_path, glob)
-                          |> Path.wildcard()
-                        end)
-                        |> Enum.concat()
-
-                      Enum.map(paths, fn path ->
-                        shortcode = Path.basename(path, Path.extname(path))
-                        external_path = Path.join("/", Path.relative_to(path, static_path))
-                        {shortcode, external_path}
-                      end)
-                    )
-
-  @emoji @finmoji_with_filenames ++ @emoji_from_globs ++ @emoji_from_file
-
-  def emojify(text, emoji \\ @emoji)
   def emojify(text, nil), do: text
 
   def emojify(text, emoji) do
@@ -160,39 +52,22 @@ defmodule Pleroma.Formatter do
   end
 
   def get_emoji(text) when is_binary(text) do
-    Enum.filter(@emoji, fn {emoji, _} -> String.contains?(text, ":#{emoji}:") end)
+    Enum.filter(Emoji.get_all(), fn {emoji, _} -> String.contains?(text, ":#{emoji}:") end)
   end
 
   def get_emoji(_), do: []
 
-  def get_custom_emoji() do
-    @emoji
-  end
-
   @link_regex ~r/[0-9a-z+\-\.]+:[0-9a-z$-_.+!*'(),]+/ui
 
-  # IANA got a list https://www.iana.org/assignments/uri-schemes/ but
-  # Stuff like ipfs isn’t in it
-  # There is very niche stuff
-  @uri_schemes [
-    "https://",
-    "http://",
-    "dat://",
-    "dweb://",
-    "gopher://",
-    "ipfs://",
-    "ipns://",
-    "irc:",
-    "ircs:",
-    "magnet:",
-    "mailto:",
-    "mumble:",
-    "ssb://",
-    "xmpp:"
-  ]
+  @uri_schemes Application.get_env(:pleroma, :uri_schemes, [])
+  @valid_schemes Keyword.get(@uri_schemes, :valid_schemes, [])
 
   # TODO: make it use something other than @link_regex
-  def html_escape(text) do
+  def html_escape(text, "text/html") do
+    HTML.filter_tags(text)
+  end
+
+  def html_escape(text, "text/plain") do
     Regex.split(@link_regex, text, include_captures: true)
     |> Enum.map_every(2, fn chunk ->
       {:safe, part} = Phoenix.HTML.html_escape(chunk)
@@ -203,14 +78,10 @@ defmodule Pleroma.Formatter do
 
   @doc "changes scheme:... urls to html links"
   def add_links({subs, text}) do
-    additionnal_schemes =
-      Application.get_env(:pleroma, :uri_schemes, [])
-      |> Keyword.get(:additionnal_schemes, [])
-
     links =
       text
       |> String.split([" ", "\t", "<br>"])
-      |> Enum.filter(fn word -> String.starts_with?(word, @uri_schemes ++ additionnal_schemes) end)
+      |> Enum.filter(fn word -> String.starts_with?(word, @valid_schemes) end)
       |> Enum.filter(fn word -> Regex.match?(@link_regex, word) end)
       |> Enum.map(fn url -> {Ecto.UUID.generate(), url} end)
       |> Enum.sort_by(fn {_, url} -> -String.length(url) end)
@@ -222,13 +93,7 @@ defmodule Pleroma.Formatter do
     subs =
       subs ++
         Enum.map(links, fn {uuid, url} ->
-          {:safe, link} = Phoenix.HTML.Link.link(url, to: url)
-
-          link =
-            link
-            |> IO.iodata_to_binary()
-
-          {uuid, link}
+          {uuid, "<a href=\"#{url}\">#{url}</a>"}
         end)
 
     {subs, uuid_text}
@@ -249,13 +114,18 @@ defmodule Pleroma.Formatter do
 
     subs =
       subs ++
-        Enum.map(mentions, fn {match, %User{ap_id: ap_id, info: info}, uuid} ->
-          ap_id = info["source_data"]["url"] || ap_id
+        Enum.map(mentions, fn {match, %User{id: id, ap_id: ap_id, info: info}, uuid} ->
+          ap_id =
+            if is_binary(info.source_data["url"]) do
+              info.source_data["url"]
+            else
+              ap_id
+            end
 
           short_match = String.split(match, "@") |> tl() |> hd()
 
           {uuid,
-           "<span><a class='mention' href='#{ap_id}'>@<span>#{short_match}</span></a></span>"}
+           "<span><a data-user='#{id}' class='mention' href='#{ap_id}'>@<span>#{short_match}</span></a></span>"}
         end)
 
     {subs, uuid_text}
@@ -277,7 +147,11 @@ defmodule Pleroma.Formatter do
     subs =
       subs ++
         Enum.map(tags, fn {tag_text, tag, uuid} ->
-          url = "<a href='#{Pleroma.Web.base_url()}/tag/#{tag}' rel='tag'>#{tag_text}</a>"
+          url =
+            "<a data-tag='#{tag}' href='#{Pleroma.Web.base_url()}/tag/#{tag}' rel='tag'>#{
+              tag_text
+            }</a>"
+
           {uuid, url}
         end)
 

lib/pleroma/gopher/server.ex

@@ -1,32 +1,33 @@
 defmodule Pleroma.Gopher.Server do
   use GenServer
   require Logger
-  @gopher Application.get_env(:pleroma, :gopher)
 
   def start_link() do
-    ip = Keyword.get(@gopher, :ip, {0, 0, 0, 0})
-    port = Keyword.get(@gopher, :port, 1234)
-    GenServer.start_link(__MODULE__, [ip, port], [])
+    config = Pleroma.Config.get(:gopher, [])
+    ip = Keyword.get(config, :ip, {0, 0, 0, 0})
+    port = Keyword.get(config, :port, 1234)
+
+    if Keyword.get(config, :enabled, false) do
+      GenServer.start_link(__MODULE__, [ip, port], [])
+    else
+      Logger.info("Gopher server disabled")
+      :ignore
+    end
   end
 
   def init([ip, port]) do
-    if Keyword.get(@gopher, :enabled, false) do
-      Logger.info("Starting gopher server on #{port}")
+    Logger.info("Starting gopher server on #{port}")
 
-      :ranch.start_listener(
-        :gopher,
-        100,
-        :ranch_tcp,
-        [port: port],
-        __MODULE__.ProtocolHandler,
-        []
-      )
+    :ranch.start_listener(
+      :gopher,
+      100,
+      :ranch_tcp,
+      [ip: ip, port: port],
+      __MODULE__.ProtocolHandler,
+      []
+    )
 
-      {:ok, %{ip: ip, port: port}}
-    else
-      Logger.info("Gopher server disabled")
-      {:ok, nil}
-    end
+    {:ok, %{ip: ip, port: port}}
   end
 end
 
@@ -37,9 +38,6 @@ defmodule Pleroma.Gopher.Server.ProtocolHandler do
   alias Pleroma.Repo
   alias Pleroma.HTML
 
-  @instance Application.get_env(:pleroma, :instance)
-  @gopher Application.get_env(:pleroma, :gopher)
-
   def start_link(ref, socket, transport, opts) do
     pid = spawn_link(__MODULE__, :init, [ref, socket, transport, opts])
     {:ok, pid}
@@ -62,7 +60,7 @@ defmodule Pleroma.Gopher.Server.ProtocolHandler do
 
   def link(name, selector, type \\ 1) do
     address = Pleroma.Web.Endpoint.host()
-    port = Keyword.get(@gopher, :port, 1234)
+    port = Pleroma.Config.get([:gopher, :port], 1234)
     "#{type}#{name}\t#{selector}\t#{address}\t#{port}\r\n"
   end
 
@@ -85,7 +83,7 @@ defmodule Pleroma.Gopher.Server.ProtocolHandler do
   end
 
   def response("") do
-    info("Welcome to #{Keyword.get(@instance, :name, "Pleroma")}!") <>
+    info("Welcome to #{Pleroma.Config.get([:instance, :name], "Pleroma")}!") <>
       link("Public Timeline", "/main/public") <>
       link("Federated Timeline", "/main/all") <> ".\r\n"
   end

lib/pleroma/html.ex

@@ -1,13 +1,28 @@
 defmodule Pleroma.HTML do
   alias HtmlSanitizeEx.Scrubber
 
-  @markup Application.get_env(:pleroma, :markup)
+  defp get_scrubbers(scrubber) when is_atom(scrubber), do: [scrubber]
+  defp get_scrubbers(scrubbers) when is_list(scrubbers), do: scrubbers
+  defp get_scrubbers(_), do: [Pleroma.HTML.Scrubber.Default]
 
-  def filter_tags(html) do
-    scrubber = Keyword.get(@markup, :scrub_policy)
+  def get_scrubbers() do
+    Pleroma.Config.get([:markup, :scrub_policy])
+    |> get_scrubbers
+  end
+
+  def filter_tags(html, nil) do
+    get_scrubbers()
+    |> Enum.reduce(html, fn scrubber, html ->
+      filter_tags(html, scrubber)
+    end)
+  end
+
+  def filter_tags(html, scrubber) do
     html |> Scrubber.scrub(scrubber)
   end
 
+  def filter_tags(html), do: filter_tags(html, nil)
+
   def strip_tags(html) do
     html |> Scrubber.scrub(Scrubber.StripTags)
   end
@@ -19,16 +34,18 @@ defmodule Pleroma.HTML.Scrubber.TwitterText do
   paragraphs, breaks and links are allowed through the filter.
   """
 
+  @markup Application.get_env(:pleroma, :markup)
+  @uri_schemes Application.get_env(:pleroma, :uri_schemes, [])
+  @valid_schemes Keyword.get(@uri_schemes, :valid_schemes, [])
+
   require HtmlSanitizeEx.Scrubber.Meta
   alias HtmlSanitizeEx.Scrubber.Meta
 
-  @valid_schemes ["http", "https"]
-
   Meta.remove_cdata_sections_before_scrub()
   Meta.strip_comments()
 
   # links
-  Meta.allow_tag_with_uri_attributes("a", ["href"], @valid_schemes)
+  Meta.allow_tag_with_uri_attributes("a", ["href", "data-user", "data-tag"], @valid_schemes)
   Meta.allow_tag_with_these_attributes("a", ["name", "title"])
 
   # paragraphs and linebreaks
@@ -39,11 +56,11 @@ defmodule Pleroma.HTML.Scrubber.TwitterText do
   Meta.allow_tag_with_these_attributes("span", [])
 
   # allow inline images for custom emoji
-  @markup Application.get_env(:pleroma, :markup)
   @allow_inline_images Keyword.get(@markup, :allow_inline_images)
 
   if @allow_inline_images do
-    Meta.allow_tag_with_uri_attributes("img", ["src"], @valid_schemes)
+    # restrict img tags to http/https only, because of MediaProxy.
+    Meta.allow_tag_with_uri_attributes("img", ["src"], ["http", "https"])
 
     Meta.allow_tag_with_these_attributes("img", [
       "width",
@@ -52,6 +69,8 @@ defmodule Pleroma.HTML.Scrubber.TwitterText do
       "alt"
     ])
   end
+
+  Meta.strip_everything_not_covered()
 end
 
 defmodule Pleroma.HTML.Scrubber.Default do
@@ -60,14 +79,18 @@ defmodule Pleroma.HTML.Scrubber.Default do
   require HtmlSanitizeEx.Scrubber.Meta
   alias HtmlSanitizeEx.Scrubber.Meta
 
-  @valid_schemes ["http", "https"]
+  @markup Application.get_env(:pleroma, :markup)
+  @uri_schemes Application.get_env(:pleroma, :uri_schemes, [])
+  @valid_schemes Keyword.get(@uri_schemes, :valid_schemes, [])
 
   Meta.remove_cdata_sections_before_scrub()
   Meta.strip_comments()
 
-  Meta.allow_tag_with_uri_attributes("a", ["href"], @valid_schemes)
+  Meta.allow_tag_with_uri_attributes("a", ["href", "data-user", "data-tag"], @valid_schemes)
   Meta.allow_tag_with_these_attributes("a", ["name", "title"])
 
+  Meta.allow_tag_with_these_attributes("abbr", ["title"])
+
   Meta.allow_tag_with_these_attributes("b", [])
   Meta.allow_tag_with_these_attributes("blockquote", [])
   Meta.allow_tag_with_these_attributes("br", [])
@@ -84,11 +107,11 @@ defmodule Pleroma.HTML.Scrubber.Default do
   Meta.allow_tag_with_these_attributes("u", [])
   Meta.allow_tag_with_these_attributes("ul", [])
 
-  @markup Application.get_env(:pleroma, :markup)
   @allow_inline_images Keyword.get(@markup, :allow_inline_images)
 
   if @allow_inline_images do
-    Meta.allow_tag_with_uri_attributes("img", ["src"], @valid_schemes)
+    # restrict img tags to http/https only, because of MediaProxy.
+    Meta.allow_tag_with_uri_attributes("img", ["src"], ["http", "https"])
 
     Meta.allow_tag_with_these_attributes("img", [
       "width",
@@ -127,3 +150,36 @@ defmodule Pleroma.HTML.Scrubber.Default do
 
   Meta.strip_everything_not_covered()
 end
+
+defmodule Pleroma.HTML.Transform.MediaProxy do
+  @moduledoc "Transforms inline image URIs to use MediaProxy."
+
+  alias Pleroma.Web.MediaProxy
+
+  def before_scrub(html), do: html
+
+  def scrub_attribute("img", {"src", "http" <> target}) do
+    media_url =
+      ("http" <> target)
+      |> MediaProxy.url()
+
+    {"src", media_url}
+  end
+
+  def scrub_attribute(_tag, attribute), do: attribute
+
+  def scrub({"img", attributes, children}) do
+    attributes =
+      attributes
+      |> Enum.map(fn attr -> scrub_attribute("img", attr) end)
+      |> Enum.reject(&is_nil(&1))
+
+    {"img", attributes, children}
+  end
+
+  def scrub({:comment, _children}), do: ""
+
+  def scrub({tag, attributes, children}), do: {tag, attributes, children}
+  def scrub({_tag, children}), do: children
+  def scrub(text), do: text
+end

lib/pleroma/http/connection.ex

@@ -0,0 +1,32 @@
+defmodule Pleroma.HTTP.Connection do
+  @moduledoc """
+  Connection for http-requests.
+  """
+
+  @hackney_options [
+    pool: :default,
+    timeout: 10000,
+    recv_timeout: 20000,
+    follow_redirect: true
+  ]
+  @adapter Application.get_env(:tesla, :adapter)
+
+  @doc """
+  Configure a client connection
+
+  # Returns
+
+  Tesla.Env.client
+  """
+  @spec new(Keyword.t()) :: Tesla.Env.client()
+  def new(opts \\ []) do
+    Tesla.client([], {@adapter, hackney_options(opts)})
+  end
+
+  # fetch Hackney options
+  #
+  defp hackney_options(opts) do
+    options = Keyword.get(opts, :adapter, [])
+    @hackney_options ++ options
+  end
+end

lib/pleroma/http/http.ex

@@ -1,14 +1,42 @@
 defmodule Pleroma.HTTP do
-  require HTTPoison
+  @moduledoc """
 
+  """
+
+  alias Pleroma.HTTP.Connection
+  alias Pleroma.HTTP.RequestBuilder, as: Builder
+
+  @doc """
+  Builds and perform http request.
+
+  # Arguments:
+  `method` - :get, :post, :put, :delete
+  `url`
+  `body`
+  `headers` - a keyworld list of headers, e.g. `[{"content-type", "text/plain"}]`
+  `options` - custom, per-request middleware or adapter options
+
+  # Returns:
+  `{:ok, %Tesla.Env{}}` or `{:error, error}`
+
+  """
   def request(method, url, body \\ "", headers \\ [], options \\ []) do
     options =
       process_request_options(options)
       |> process_sni_options(url)
 
-    HTTPoison.request(method, url, body, headers, options)
+    %{}
+    |> Builder.method(method)
+    |> Builder.headers(headers)
+    |> Builder.opts(options)
+    |> Builder.url(url)
+    |> Builder.add_param(:body, :body, body)
+    |> Enum.into([])
+    |> (&Tesla.request(Connection.new(), &1)).()
   end
 
+  defp process_sni_options(options, nil), do: options
+
   defp process_sni_options(options, url) do
     uri = URI.parse(url)
     host = uri.host |> to_charlist()
@@ -22,6 +50,7 @@ defmodule Pleroma.HTTP do
   def process_request_options(options) do
     config = Application.get_env(:pleroma, :http, [])
     proxy = Keyword.get(config, :proxy_url, nil)
+    options = options ++ [adapter: [pool: :default]]
 
     case proxy do
       nil -> options
@@ -29,8 +58,19 @@ defmodule Pleroma.HTTP do
     end
   end
 
-  def get(url, headers \\ [], options \\ []), do: request(:get, url, "", headers, options)
+  @doc """
+  Performs GET request.
 
+  See `Pleroma.HTTP.request/5`
+  """
+  def get(url, headers \\ [], options \\ []),
+    do: request(:get, url, "", headers, options)
+
+  @doc """
+  Performs POST request.
+
+  See `Pleroma.HTTP.request/5`
+  """
   def post(url, body, headers \\ [], options \\ []),
     do: request(:post, url, body, headers, options)
 end

lib/pleroma/http/request_builder.ex

@@ -0,0 +1,126 @@
+defmodule Pleroma.HTTP.RequestBuilder do
+  @moduledoc """
+  Helper functions for building Tesla requests
+  """
+
+  @doc """
+  Specify the request method when building a request
+
+  ## Parameters
+
+  - request (Map) - Collected request options
+  - m (atom) - Request method
+
+  ## Returns
+
+  Map
+  """
+  @spec method(map(), atom) :: map()
+  def method(request, m) do
+    Map.put_new(request, :method, m)
+  end
+
+  @doc """
+  Specify the request method when building a request
+
+  ## Parameters
+
+  - request (Map) - Collected request options
+  - u (String) - Request URL
+
+  ## Returns
+
+  Map
+  """
+  @spec url(map(), String.t()) :: map()
+  def url(request, u) do
+    Map.put_new(request, :url, u)
+  end
+
+  @doc """
+  Add headers to the request
+  """
+  @spec headers(map(), list(tuple)) :: map()
+  def headers(request, h) do
+    Map.put_new(request, :headers, h)
+  end
+
+  @doc """
+  Add custom, per-request middleware or adapter options to the request
+  """
+  @spec opts(map(), Keyword.t()) :: map()
+  def opts(request, options) do
+    Map.put_new(request, :opts, options)
+  end
+
+  @doc """
+  Add optional parameters to the request
+
+  ## Parameters
+
+  - request (Map) - Collected request options
+  - definitions (Map) - Map of parameter name to parameter location.
+  - options (KeywordList) - The provided optional parameters
+
+  ## Returns
+
+  Map
+  """
+  @spec add_optional_params(map(), %{optional(atom) => atom}, keyword()) :: map()
+  def add_optional_params(request, _, []), do: request
+
+  def add_optional_params(request, definitions, [{key, value} | tail]) do
+    case definitions do
+      %{^key => location} ->
+        request
+        |> add_param(location, key, value)
+        |> add_optional_params(definitions, tail)
+
+      _ ->
+        add_optional_params(request, definitions, tail)
+    end
+  end
+
+  @doc """
+  Add optional parameters to the request
+
+  ## Parameters
+
+  - request (Map) - Collected request options
+  - location (atom) - Where to put the parameter
+  - key (atom) - The name of the parameter
+  - value (any) - The value of the parameter
+
+  ## Returns
+
+  Map
+  """
+  @spec add_param(map(), atom, atom, any()) :: map()
+  def add_param(request, :body, :body, value), do: Map.put(request, :body, value)
+
+  def add_param(request, :body, key, value) do
+    request
+    |> Map.put_new_lazy(:body, &Tesla.Multipart.new/0)
+    |> Map.update!(
+      :body,
+      &Tesla.Multipart.add_field(&1, key, Poison.encode!(value),
+        headers: [{:"Content-Type", "application/json"}]
+      )
+    )
+  end
+
+  def add_param(request, :file, name, path) do
+    request
+    |> Map.put_new_lazy(:body, &Tesla.Multipart.new/0)
+    |> Map.update!(:body, &Tesla.Multipart.add_file(&1, path, name: name))
+  end
+
+  def add_param(request, :form, name, value) do
+    request
+    |> Map.update(:body, %{name => value}, &Map.put(&1, name, value))
+  end
+
+  def add_param(request, location, key, value) do
+    Map.update(request, location, [{key, value}], &(&1 ++ [{key, value}]))
+  end
+end

lib/pleroma/list.ex

@@ -23,7 +23,7 @@ defmodule Pleroma.List do
     |> validate_required([:following])
   end
 
-  def for_user(user, opts) do
+  def for_user(user, _opts) do
     query =
       from(
         l in Pleroma.List,
@@ -46,7 +46,7 @@ defmodule Pleroma.List do
     Repo.one(query)
   end
 
-  def get_following(%Pleroma.List{following: following} = list) do
+  def get_following(%Pleroma.List{following: following} = _list) do
     q =
       from(
         u in User,
@@ -69,6 +69,25 @@ defmodule Pleroma.List do
     Repo.all(query)
   end
 
+  # Get lists to which the account belongs.
+  def get_lists_account_belongs(%User{} = owner, account_id) do
+    user = Repo.get(User, account_id)
+
+    query =
+      from(
+        l in Pleroma.List,
+        where:
+          l.user_id == ^owner.id and
+            fragment(
+              "? = ANY(?)",
+              ^user.follower_address,
+              l.following
+            )
+      )
+
+    Repo.all(query)
+  end
+
   def rename(%Pleroma.List{} = list, title) do
     list
     |> title_changeset(%{title: title})

lib/pleroma/mime.ex

@@ -0,0 +1,108 @@
+defmodule Pleroma.MIME do
+  @moduledoc """
+  Returns the mime-type of a binary and optionally a normalized file-name.
+  """
+  @default "application/octet-stream"
+  @read_bytes 35
+
+  @spec file_mime_type(String.t()) ::
+          {:ok, content_type :: String.t(), filename :: String.t()} | {:error, any()} | :error
+  def file_mime_type(path, filename) do
+    with {:ok, content_type} <- file_mime_type(path),
+         filename <- fix_extension(filename, content_type) do
+      {:ok, content_type, filename}
+    end
+  end
+
+  @spec file_mime_type(String.t()) :: {:ok, String.t()} | {:error, any()} | :error
+  def file_mime_type(filename) do
+    File.open(filename, [:read], fn f ->
+      check_mime_type(IO.binread(f, @read_bytes))
+    end)
+  end
+
+  def bin_mime_type(binary, filename) do
+    with {:ok, content_type} <- bin_mime_type(binary),
+         filename <- fix_extension(filename, content_type) do
+      {:ok, content_type, filename}
+    end
+  end
+
+  @spec bin_mime_type(binary()) :: {:ok, String.t()} | :error
+  def bin_mime_type(<<head::binary-size(@read_bytes), _::binary>>) do
+    {:ok, check_mime_type(head)}
+  end
+
+  def bin_mime_type(_), do: :error
+
+  def mime_type(<<_::binary>>), do: {:ok, @default}
+
+  defp fix_extension(filename, content_type) do
+    parts = String.split(filename, ".")
+
+    new_filename =
+      if length(parts) > 1 do
+        Enum.drop(parts, -1) |> Enum.join(".")
+      else
+        Enum.join(parts)
+      end
+
+    cond do
+      content_type == "application/octet-stream" ->
+        filename
+
+      ext = List.first(MIME.extensions(content_type)) ->
+        new_filename <> "." <> ext
+
+      true ->
+        Enum.join([new_filename, String.split(content_type, "/") |> List.last()], ".")
+    end
+  end
+
+  defp check_mime_type(<<0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, _::binary>>) do
+    "image/png"
+  end
+
+  defp check_mime_type(<<0x47, 0x49, 0x46, 0x38, _, 0x61, _::binary>>) do
+    "image/gif"
+  end
+
+  defp check_mime_type(<<0xFF, 0xD8, 0xFF, _::binary>>) do
+    "image/jpeg"
+  end
+
+  defp check_mime_type(<<0x1A, 0x45, 0xDF, 0xA3, _::binary>>) do
+    "video/webm"
+  end
+
+  defp check_mime_type(<<0x00, 0x00, 0x00, _, 0x66, 0x74, 0x79, 0x70, _::binary>>) do
+    "video/mp4"
+  end
+
+  defp check_mime_type(<<0x49, 0x44, 0x33, _::binary>>) do
+    "audio/mpeg"
+  end
+
+  defp check_mime_type(<<255, 251, _, 68, 0, 0, 0, 0, _::binary>>) do
+    "audio/mpeg"
+  end
+
+  defp check_mime_type(
+         <<0x4F, 0x67, 0x67, 0x53, 0x00, 0x02, 0x00, 0x00, _::size(160), 0x80, 0x74, 0x68, 0x65,
+           0x6F, 0x72, 0x61, _::binary>>
+       ) do
+    "video/ogg"
+  end
+
+  defp check_mime_type(<<0x4F, 0x67, 0x67, 0x53, 0x00, 0x02, 0x00, 0x00, _::binary>>) do
+    "audio/ogg"
+  end
+
+  defp check_mime_type(<<0x52, 0x49, 0x46, 0x46, _::binary>>) do
+    "audio/wav"
+  end
+
+  defp check_mime_type(_) do
+    @default
+  end
+end

lib/pleroma/notification.ex

@@ -1,6 +1,6 @@
 defmodule Pleroma.Notification do
   use Ecto.Schema
-  alias Pleroma.{User, Activity, Notification, Repo}
+  alias Pleroma.{User, Activity, Notification, Repo, Object}
   import Ecto.Query
 
   schema "notifications" do
@@ -42,6 +42,20 @@ defmodule Pleroma.Notification do
     Repo.all(query)
   end
 
+  def set_read_up_to(%{id: user_id} = _user, id) do
+    query =
+      from(
+        n in Notification,
+        where: n.user_id == ^user_id,
+        where: n.id <= ^id,
+        update: [
+          set: [seen: true]
+        ]
+      )
+
+    Repo.update_all(query, [])
+  end
+
   def get(%{id: user_id} = _user, id) do
     query =
       from(
@@ -81,7 +95,7 @@ defmodule Pleroma.Notification do
 
   def create_notifications(%Activity{id: _, data: %{"to" => _, "type" => type}} = activity)
       when type in ["Create", "Like", "Announce", "Follow"] do
-    users = User.get_notified_from_activity(activity)
+    users = get_notified_from_activity(activity)
 
     notifications = Enum.map(users, fn user -> create_notification(activity, user) end)
     {:ok, notifications}
@@ -96,7 +110,68 @@ defmodule Pleroma.Notification do
       notification = %Notification{user_id: user.id, activity: activity}
       {:ok, notification} = Repo.insert(notification)
       Pleroma.Web.Streamer.stream("user", notification)
+      Pleroma.Web.Push.send(notification)
       notification
     end
   end
+
+  def get_notified_from_activity(activity, local_only \\ true)
+
+  def get_notified_from_activity(
+        %Activity{data: %{"to" => _, "type" => type} = _data} = activity,
+        local_only
+      )
+      when type in ["Create", "Like", "Announce", "Follow"] do
+    recipients =
+      []
+      |> maybe_notify_to_recipients(activity)
+      |> maybe_notify_mentioned_recipients(activity)
+      |> Enum.uniq()
+
+    User.get_users_from_set(recipients, local_only)
+  end
+
+  def get_notified_from_activity(_, _local_only), do: []
+
+  defp maybe_notify_to_recipients(
+         recipients,
+         %Activity{data: %{"to" => to, "type" => _type}} = _activity
+       ) do
+    recipients ++ to
+  end
+
+  defp maybe_notify_mentioned_recipients(
+         recipients,
+         %Activity{data: %{"to" => _to, "type" => type} = data} = _activity
+       )
+       when type == "Create" do
+    object = Object.normalize(data["object"])
+
+    object_data =
+      cond do
+        !is_nil(object) ->
+          object.data
+
+        is_map(data["object"]) ->
+          data["object"]
+
+        true ->
+          %{}
+      end
+
+    tagged_mentions = maybe_extract_mentions(object_data)
+
+    recipients ++ tagged_mentions
+  end
+
+  defp maybe_notify_mentioned_recipients(recipients, _), do: recipients
+
+  defp maybe_extract_mentions(%{"tag" => tag}) do
+    tag
+    |> Enum.filter(fn x -> is_map(x) end)
+    |> Enum.filter(fn x -> x["type"] == "Mention" end)
+    |> Enum.map(fn x -> x["href"] end)
+  end
+
+  defp maybe_extract_mentions(_), do: []
 end

lib/pleroma/object.ex

@@ -1,6 +1,6 @@
 defmodule Pleroma.Object do
   use Ecto.Schema
-  alias Pleroma.{Repo, Object}
+  alias Pleroma.{Repo, Object, User, Activity}
   import Ecto.{Query, Changeset}
 
   schema "objects" do
@@ -31,13 +31,22 @@ defmodule Pleroma.Object do
   def normalize(ap_id) when is_binary(ap_id), do: Object.get_by_ap_id(ap_id)
   def normalize(_), do: nil
 
-  def get_cached_by_ap_id(ap_id) do
-    if Mix.env() == :test do
+  # Owned objects can only be mutated by their owner
+  def authorize_mutation(%Object{data: %{"actor" => actor}}, %User{ap_id: ap_id}),
+    do: actor == ap_id
+
+  # Legacy objects can be mutated by anybody
+  def authorize_mutation(%Object{}, %User{}), do: true
+
+  if Mix.env() == :test do
+    def get_cached_by_ap_id(ap_id) do
       get_by_ap_id(ap_id)
-    else
+    end
+  else
+    def get_cached_by_ap_id(ap_id) do
       key = "object:#{ap_id}"
 
-      Cachex.fetch!(:user_cache, key, fn _ ->
+      Cachex.fetch!(:object_cache, key, fn _ ->
         object = get_by_ap_id(ap_id)
 
         if object do
@@ -52,4 +61,12 @@ defmodule Pleroma.Object do
   def context_mapping(context) do
     Object.change(%Object{}, %{data: %{"id" => context}})
   end
+
+  def delete(%Object{data: %{"id" => id}} = object) do
+    with Repo.delete(object),
+         Repo.delete_all(Activity.all_non_create_by_object_ap_id_q(id)),
+         {:ok, true} <- Cachex.del(:object_cache, "object:#{id}") do
+      {:ok, object}
+    end
+  end
 end

lib/pleroma/plugs/authentication_plug.ex

@@ -26,14 +26,7 @@ defmodule Pleroma.Plugs.AuthenticationPlug do
     end
   end
 
-  def call(
-        %{
-          assigns: %{
-            auth_credentials: %{password: password}
-          }
-        } = conn,
-        _
-      ) do
+  def call(%{assigns: %{auth_credentials: %{password: _}}} = conn, _) do
     Pbkdf2.dummy_checkpw()
     conn
   end

lib/pleroma/plugs/basic_auth_decoder_plug.ex

@@ -5,7 +5,7 @@ defmodule Pleroma.Plugs.BasicAuthDecoderPlug do
     options
   end
 
-  def call(conn, opts) do
+  def call(conn, _opts) do
     with ["Basic " <> header] <- get_req_header(conn, "authorization"),
          {:ok, userinfo} <- Base.decode64(header),
          [username, password] <- String.split(userinfo, ":", parts: 2) do

lib/pleroma/plugs/federating_plug.ex

@@ -0,0 +1,18 @@
+defmodule Pleroma.Web.FederatingPlug do
+  import Plug.Conn
+
+  def init(options) do
+    options
+  end
+
+  def call(conn, _opts) do
+    if Keyword.get(Application.get_env(:pleroma, :instance), :federating) do
+      conn
+    else
+      conn
+      |> put_status(404)
+      |> Phoenix.Controller.render(Pleroma.Web.ErrorView, "404.json")
+      |> halt()
+    end
+  end
+end

lib/pleroma/plugs/http_security_plug.ex

@@ -0,0 +1,63 @@
+defmodule Pleroma.Plugs.HTTPSecurityPlug do
+  alias Pleroma.Config
+  import Plug.Conn
+
+  def init(opts), do: opts
+
+  def call(conn, _options) do
+    if Config.get([:http_security, :enabled]) do
+      conn
+      |> merge_resp_headers(headers())
+      |> maybe_send_sts_header(Config.get([:http_security, :sts]))
+    else
+      conn
+    end
+  end
+
+  defp headers do
+    referrer_policy = Config.get([:http_security, :referrer_policy])
+
+    [
+      {"x-xss-protection", "1; mode=block"},
+      {"x-permitted-cross-domain-policies", "none"},
+      {"x-frame-options", "DENY"},
+      {"x-content-type-options", "nosniff"},
+      {"referrer-policy", referrer_policy},
+      {"x-download-options", "noopen"},
+      {"content-security-policy", csp_string() <> ";"}
+    ]
+  end
+
+  defp csp_string do
+    protocol = Config.get([Pleroma.Web.Endpoint, :protocol])
+
+    [
+      "default-src 'none'",
+      "base-uri 'self'",
+      "frame-ancestors 'none'",
+      "img-src 'self' data: https:",
+      "media-src 'self' https:",
+      "style-src 'self' 'unsafe-inline'",
+      "font-src 'self'",
+      "script-src 'self'",
+      "connect-src 'self' " <> String.replace(Pleroma.Web.Endpoint.static_url(), "http", "ws"),
+      "manifest-src 'self'",
+      if protocol == "https" do
+        "upgrade-insecure-requests"
+      end
+    ]
+    |> Enum.join("; ")
+  end
+
+  defp maybe_send_sts_header(conn, true) do
+    max_age_sts = Config.get([:http_security, :sts_max_age])
+    max_age_ct = Config.get([:http_security, :ct_max_age])
+
+    merge_resp_headers(conn, [
+      {"strict-transport-security", "max-age=#{max_age_sts}; includeSubDomains"},
+      {"expect-ct", "enforce, max-age=#{max_age_ct}"}
+    ])
+  end
+
+  defp maybe_send_sts_header(conn, _), do: conn
+end

lib/pleroma/plugs/oauth_plug.ex

@@ -1,30 +1,70 @@
 defmodule Pleroma.Plugs.OAuthPlug do
   import Plug.Conn
-  alias Pleroma.User
-  alias Pleroma.Repo
-  alias Pleroma.Web.OAuth.Token
+  import Ecto.Query
 
-  def init(options) do
-    options
-  end
+  alias Pleroma.{
+    User,
+    Repo,
+    Web.OAuth.Token
+  }
+
+  @realm_reg Regex.compile!("Bearer\:?\s+(.*)$", "i")
+
+  def init(options), do: options
 
   def call(%{assigns: %{user: %User{}}} = conn, _), do: conn
 
   def call(conn, _) do
-    token =
-      case get_req_header(conn, "authorization") do
-        ["Bearer " <> header] -> header
-        _ -> get_session(conn, :oauth_token)
-      end
-
-    with token when not is_nil(token) <- token,
-         %Token{user_id: user_id} <- Repo.get_by(Token, token: token),
-         %User{} = user <- Repo.get(User, user_id),
-         false <- !!user.info["deactivated"] do
+    with {:ok, token} <- fetch_token(conn),
+         {:ok, user} <- fetch_user(token) do
       conn
+      |> assign(:token, token)
       |> assign(:user, user)
     else
       _ -> conn
     end
   end
+
+  # Gets user by token
+  #
+  @spec fetch_user(String.t()) :: {:ok, User.t()} | nil
+  defp fetch_user(token) do
+    query = from(q in Token, where: q.token == ^token, preload: [:user])
+
+    with %Token{user: %{info: %{deactivated: false} = _} = user} <- Repo.one(query) do
+      {:ok, user}
+    end
+  end
+
+  # Gets token from session by :oauth_token key
+  #
+  @spec fetch_token_from_session(Plug.Conn.t()) :: :no_token_found | {:ok, String.t()}
+  defp fetch_token_from_session(conn) do
+    case get_session(conn, :oauth_token) do
+      nil -> :no_token_found
+      token -> {:ok, token}
+    end
+  end
+
+  # Gets token from headers
+  #
+  @spec fetch_token(Plug.Conn.t()) :: :no_token_found | {:ok, String.t()}
+  defp fetch_token(%Plug.Conn{} = conn) do
+    headers = get_req_header(conn, "authorization")
+
+    with :no_token_found <- fetch_token(headers),
+         do: fetch_token_from_session(conn)
+  end
+
+  @spec fetch_token(Keyword.t()) :: :no_token_found | {:ok, String.t()}
+  defp fetch_token([]), do: :no_token_found
+
+  defp fetch_token([token | tail]) do
+    trimmed_token = String.trim(token)
+
+    case Regex.run(@realm_reg, trimmed_token) do
+      [_, match] -> {:ok, String.trim(match)}
+      _ -> fetch_token(tail)
+    end
+  end
 end

lib/pleroma/plugs/session_authentication_plug.ex

@@ -1,6 +1,5 @@
 defmodule Pleroma.Plugs.SessionAuthenticationPlug do
   import Plug.Conn
-  alias Pleroma.User
 
   def init(options) do
     options

lib/pleroma/plugs/uploaded_media.ex

@@ -0,0 +1,74 @@
+defmodule Pleroma.Plugs.UploadedMedia do
+  @moduledoc """
+  """
+
+  import Plug.Conn
+  require Logger
+
+  @behaviour Plug
+  # no slashes
+  @path "media"
+
+  def init(_opts) do
+    static_plug_opts =
+      []
+      |> Keyword.put(:from, "__unconfigured_media_plug")
+      |> Keyword.put(:at, "/__unconfigured_media_plug")
+      |> Plug.Static.init()
+
+    %{static_plug_opts: static_plug_opts}
+  end
+
+  def call(conn = %{request_path: <<"/", @path, "/", file::binary>>}, opts) do
+    config = Pleroma.Config.get([Pleroma.Upload])
+
+    with uploader <- Keyword.fetch!(config, :uploader),
+         proxy_remote = Keyword.get(config, :proxy_remote, false),
+         {:ok, get_method} <- uploader.get_file(file) do
+      get_media(conn, get_method, proxy_remote, opts)
+    else
+      _ ->
+        conn
+        |> send_resp(500, "Failed")
+        |> halt()
+    end
+  end
+
+  def call(conn, _opts), do: conn
+
+  defp get_media(conn, {:static_dir, directory}, _, opts) do
+    static_opts =
+      Map.get(opts, :static_plug_opts)
+      |> Map.put(:at, [@path])
+      |> Map.put(:from, directory)
+
+    conn = Plug.Static.call(conn, static_opts)
+
+    if conn.halted do
+      conn
+    else
+      conn
+      |> send_resp(404, "Not found")
+      |> halt()
+    end
+  end
+
+  defp get_media(conn, {:url, url}, true, _) do
+    conn
+    |> Pleroma.ReverseProxy.call(url, Pleroma.Config.get([Pleroma.Upload, :proxy_opts], []))
+  end
+
+  defp get_media(conn, {:url, url}, _, _) do
+    conn
+    |> Phoenix.Controller.redirect(external: url)
+    |> halt()
+  end
+
+  defp get_media(conn, unknown, _, _) do
+    Logger.error("#{__MODULE__}: Unknown get startegy: #{inspect(unknown)}")
+
+    conn
+    |> send_resp(500, "Internal Error")
+    |> halt()
+  end
+end

lib/pleroma/plugs/user_enabled_plug.ex

@@ -6,7 +6,7 @@ defmodule Pleroma.Plugs.UserEnabledPlug do
     options
   end
 
-  def call(%{assigns: %{user: %User{info: %{"deactivated" => true}}}} = conn, _) do
+  def call(%{assigns: %{user: %User{info: %{deactivated: true}}}} = conn, _) do
     conn
     |> assign(:user, nil)
   end

lib/pleroma/plugs/user_fetcher_plug.ex

@@ -7,7 +7,7 @@ defmodule Pleroma.Plugs.UserFetcherPlug do
     options
   end
 
-  def call(conn, options) do
+  def call(conn, _options) do
     with %{auth_credentials: %{username: username}} <- conn.assigns,
          {:ok, %User{} = user} <- user_fetcher(username) do
       conn

lib/pleroma/plugs/user_is_admin_plug.ex

@@ -0,0 +1,19 @@
+defmodule Pleroma.Plugs.UserIsAdminPlug do
+  import Plug.Conn
+  alias Pleroma.User
+
+  def init(options) do
+    options
+  end
+
+  def call(%{assigns: %{user: %User{info: %{is_admin: true}}}} = conn, _) do
+    conn
+  end
+
+  def call(conn, _) do
+    conn
+    |> put_resp_content_type("application/json")
+    |> send_resp(403, Jason.encode!(%{error: "User is not admin."}))
+    |> halt
+  end
+end

lib/pleroma/reverse_proxy.ex

@@ -0,0 +1,344 @@
+defmodule Pleroma.ReverseProxy do
+  @keep_req_headers ~w(accept user-agent accept-encoding cache-control if-modified-since if-unmodified-since if-none-match if-range range)
+  @resp_cache_headers ~w(etag date last-modified cache-control)
+  @keep_resp_headers @resp_cache_headers ++
+                       ~w(content-type content-disposition content-encoding content-range accept-ranges vary)
+  @default_cache_control_header "public, max-age=1209600"
+  @valid_resp_codes [200, 206, 304]
+  @max_read_duration :timer.seconds(30)
+  @max_body_length :infinity
+  @methods ~w(GET HEAD)
+
+  @moduledoc """
+  A reverse proxy.
+
+      Pleroma.ReverseProxy.call(conn, url, options)
+
+  It is not meant to be added into a plug pipeline, but to be called from another plug or controller.
+
+  Supports `#{inspect(@methods)}` HTTP methods, and only allows `#{inspect(@valid_resp_codes)}` status codes.
+
+  Responses are chunked to the client while downloading from the upstream.
+
+  Some request / responses headers are preserved:
+
+  * request: `#{inspect(@keep_req_headers)}`
+  * response: `#{inspect(@keep_resp_headers)}`
+
+  If no caching headers (`#{inspect(@resp_cache_headers)}`) are returned by upstream, `cache-control` will be
+  set to `#{inspect(@default_cache_control_header)}`.
+
+  Options:
+
+  * `redirect_on_failure` (default `false`). Redirects the client to the real remote URL if there's any HTTP
+  errors. Any error during body processing will not be redirected as the response is chunked. This may expose
+  remote URL, clients IPs, ….
+
+  * `max_body_length` (default `#{inspect(@max_body_length)}`): limits the content length to be approximately the
+  specified length. It is validated with the `content-length` header and also verified when proxying.
+
+  * `max_read_duration` (default `#{inspect(@max_read_duration)}` ms): the total time the connection is allowed to
+  read from the remote upstream.
+
+  * `inline_content_types`:
+    * `true` will not alter `content-disposition` (up to the upstream),
+    * `false` will add `content-disposition: attachment` to any request,
+    * a list of whitelisted content types
+
+    * `keep_user_agent` will forward the client's user-agent to the upstream. This may be useful if the upstream is
+    doing content transformation (encoding, …) depending on the request.
+
+  * `req_headers`, `resp_headers` additional headers.
+
+  * `http`: options for [hackney](https://github.com/benoitc/hackney).
+
+  """
+  @hackney Application.get_env(:pleroma, :hackney, :hackney)
+  @httpoison Application.get_env(:pleroma, :httpoison, HTTPoison)
+
+  @default_hackney_options []
+
+  @inline_content_types [
+    "image/gif",
+    "image/jpeg",
+    "image/jpg",
+    "image/png",
+    "image/svg+xml",
+    "audio/mpeg",
+    "audio/mp3",
+    "video/webm",
+    "video/mp4",
+    "video/quicktime"
+  ]
+
+  require Logger
+  import Plug.Conn
+
+  @type option() ::
+          {:keep_user_agent, boolean}
+          | {:max_read_duration, :timer.time() | :infinity}
+          | {:max_body_length, non_neg_integer() | :infinity}
+          | {:http, []}
+          | {:req_headers, [{String.t(), String.t()}]}
+          | {:resp_headers, [{String.t(), String.t()}]}
+          | {:inline_content_types, boolean() | [String.t()]}
+          | {:redirect_on_failure, boolean()}
+
+  @spec call(Plug.Conn.t(), url :: String.t(), [option()]) :: Plug.Conn.t()
+  def call(_conn, _url, _opts \\ [])
+
+  def call(conn = %{method: method}, url, opts) when method in @methods do
+    hackney_opts =
+      @default_hackney_options
+      |> Keyword.merge(Keyword.get(opts, :http, []))
+      |> @httpoison.process_request_options()
+
+    req_headers = build_req_headers(conn.req_headers, opts)
+
+    opts =
+      if filename = Pleroma.Web.MediaProxy.filename(url) do
+        Keyword.put_new(opts, :attachment_name, filename)
+      else
+        opts
+      end
+
+    with {:ok, code, headers, client} <- request(method, url, req_headers, hackney_opts),
+         :ok <- header_length_constraint(headers, Keyword.get(opts, :max_body_length)) do
+      response(conn, client, url, code, headers, opts)
+    else
+      {:ok, code, headers} ->
+        head_response(conn, url, code, headers, opts)
+        |> halt()
+
+      {:error, {:invalid_http_response, code}} ->
+        Logger.error("#{__MODULE__}: request to #{inspect(url)} failed with HTTP status #{code}")
+
+        conn
+        |> error_or_redirect(
+          url,
+          code,
+          "Request failed: " <> Plug.Conn.Status.reason_phrase(code),
+          opts
+        )
+        |> halt()
+
+      {:error, error} ->
+        Logger.error("#{__MODULE__}: request to #{inspect(url)} failed: #{inspect(error)}")
+
+        conn
+        |> error_or_redirect(url, 500, "Request failed", opts)
+        |> halt()
+    end
+  end
+
+  def call(conn, _, _) do
+    conn
+    |> send_resp(400, Plug.Conn.Status.reason_phrase(400))
+    |> halt()
+  end
+
+  defp request(method, url, headers, hackney_opts) do
+    Logger.debug("#{__MODULE__} #{method} #{url} #{inspect(headers)}")
+    method = method |> String.downcase() |> String.to_existing_atom()
+
+    case @hackney.request(method, url, headers, "", hackney_opts) do
+      {:ok, code, headers, client} when code in @valid_resp_codes ->
+        {:ok, code, downcase_headers(headers), client}
+
+      {:ok, code, headers} when code in @valid_resp_codes ->
+        {:ok, code, downcase_headers(headers)}
+
+      {:ok, code, _, _} ->
+        {:error, {:invalid_http_response, code}}
+
+      {:error, error} ->
+        {:error, error}
+    end
+  end
+
+  defp response(conn, client, url, status, headers, opts) do
+    result =
+      conn
+      |> put_resp_headers(build_resp_headers(headers, opts))
+      |> send_chunked(status)
+      |> chunk_reply(client, opts)
+
+    case result do
+      {:ok, conn} ->
+        halt(conn)
+
+      {:error, :closed, conn} ->
+        :hackney.close(client)
+        halt(conn)
+
+      {:error, error, conn} ->
+        Logger.warn(
+          "#{__MODULE__} request to #{url} failed while reading/chunking: #{inspect(error)}"
+        )
+
+        :hackney.close(client)
+        halt(conn)
+    end
+  end
+
+  defp chunk_reply(conn, client, opts) do
+    chunk_reply(conn, client, opts, 0, 0)
+  end
+
+  defp chunk_reply(conn, client, opts, sent_so_far, duration) do
+    with {:ok, duration} <-
+           check_read_duration(
+             duration,
+             Keyword.get(opts, :max_read_duration, @max_read_duration)
+           ),
+         {:ok, data} <- @hackney.stream_body(client),
+         {:ok, duration} <- increase_read_duration(duration),
+         sent_so_far = sent_so_far + byte_size(data),
+         :ok <- body_size_constraint(sent_so_far, Keyword.get(opts, :max_body_size)),
+         {:ok, conn} <- chunk(conn, data) do
+      chunk_reply(conn, client, opts, sent_so_far, duration)
+    else
+      :done -> {:ok, conn}
+      {:error, error} -> {:error, error, conn}
+    end
+  end
+
+  defp head_response(conn, _url, code, headers, opts) do
+    conn
+    |> put_resp_headers(build_resp_headers(headers, opts))
+    |> send_resp(code, "")
+  end
+
+  defp error_or_redirect(conn, url, code, body, opts) do
+    if Keyword.get(opts, :redirect_on_failure, false) do
+      conn
+      |> Phoenix.Controller.redirect(external: url)
+      |> halt()
+    else
+      conn
+      |> send_resp(code, body)
+      |> halt
+    end
+  end
+
+  defp downcase_headers(headers) do
+    Enum.map(headers, fn {k, v} ->
+      {String.downcase(k), v}
+    end)
+  end
+
+  defp get_content_type(headers) do
+    {_, content_type} =
+      List.keyfind(headers, "content-type", 0, {"content-type", "application/octet-stream"})
+
+    [content_type | _] = String.split(content_type, ";")
+    content_type
+  end
+
+  defp put_resp_headers(conn, headers) do
+    Enum.reduce(headers, conn, fn {k, v}, conn ->
+      put_resp_header(conn, k, v)
+    end)
+  end
+
+  defp build_req_headers(headers, opts) do
+    headers
+    |> downcase_headers()
+    |> Enum.filter(fn {k, _} -> k in @keep_req_headers end)
+    |> (fn headers ->
+          headers = headers ++ Keyword.get(opts, :req_headers, [])
+
+          if Keyword.get(opts, :keep_user_agent, false) do
+            List.keystore(
+              headers,
+              "user-agent",
+              0,
+              {"user-agent", Pleroma.Application.user_agent()}
+            )
+          else
+            headers
+          end
+        end).()
+  end
+
+  defp build_resp_headers(headers, opts) do
+    headers
+    |> Enum.filter(fn {k, _} -> k in @keep_resp_headers end)
+    |> build_resp_cache_headers(opts)
+    |> build_resp_content_disposition_header(opts)
+    |> (fn headers -> headers ++ Keyword.get(opts, :resp_headers, []) end).()
+  end
+
+  defp build_resp_cache_headers(headers, _opts) do
+    has_cache? = Enum.any?(headers, fn {k, _} -> k in @resp_cache_headers end)
+
+    if has_cache? do
+      headers
+    else
+      List.keystore(headers, "cache-control", 0, {"cache-control", @default_cache_control_header})
+    end
+  end
+
+  defp build_resp_content_disposition_header(headers, opts) do
+    opt = Keyword.get(opts, :inline_content_types, @inline_content_types)
+
+    content_type = get_content_type(headers)
+
+    attachment? =
+      cond do
+        is_list(opt) && !Enum.member?(opt, content_type) -> true
+        opt == false -> true
+        true -> false
+      end
+
+    if attachment? do
+      disposition = "attachment; filename=" <> Keyword.get(opts, :attachment_name, "attachment")
+      List.keystore(headers, "content-disposition", 0, {"content-disposition", disposition})
+    else
+      headers
+    end
+  end
+
+  defp header_length_constraint(headers, limit) when is_integer(limit) and limit > 0 do
+    with {_, size} <- List.keyfind(headers, "content-length", 0),
+         {size, _} <- Integer.parse(size),
+         true <- size <= limit do
+      :ok
+    else
+      false ->
+        {:error, :body_too_large}
+
+      _ ->
+        :ok
+    end
+  end
+
+  defp header_length_constraint(_, _), do: :ok
+
+  defp body_size_constraint(size, limit) when is_integer(limit) and limit > 0 and size >= limit do
+    {:error, :body_too_large}
+  end
+
+  defp body_size_constraint(_, _), do: :ok
+
+  defp check_read_duration(duration, max)
+       when is_integer(duration) and is_integer(max) and max > 0 do
+    if duration > max do
+      {:error, :read_duration_exceeded}
+    else
+      {:ok, {duration, :erlang.system_time(:millisecond)}}
+    end
+  end
+
+  defp check_read_duration(_, _), do: {:ok, :no_duration_limit, :no_duration_limit}
+
+  defp increase_read_duration({previous_duration, started})
+       when is_integer(previous_duration) and is_integer(started) do
+    duration = :erlang.system_time(:millisecond) - started
+    {:ok, previous_duration + duration}
+  end
+
+  defp increase_read_duration(_) do
+    {:ok, :no_duration_limit, :no_duration_limit}
+  end
+end

lib/pleroma/upload.ex

@@ -1,71 +1,208 @@
 defmodule Pleroma.Upload do
+  @moduledoc """
+  # Upload
+
+  Options:
+  * `:type`: presets for activity type (defaults to Document) and size limits from app configuration
+  * `:description`: upload alternative text
+  * `:base_url`: override base url
+  * `:uploader`: override uploader
+  * `:filters`: override filters
+  * `:size_limit`: override size limit
+  * `:activity_type`: override activity type
+
+  The `%Pleroma.Upload{}` struct: all documented fields are meant to be overwritten in filters:
+
+  * `:id` - the upload id.
+  * `:name` - the upload file name.
+  * `:path` - the upload path: set at first to `id/name` but can be changed. Keep in mind that the path
+    is once created permanent and changing it (especially in uploaders) is probably a bad idea!
+  * `:tempfile` - path to the temporary file. Prefer in-place changes on the file rather than changing the
+  path as the temporary file is also tracked by `Plug.Upload{}` and automatically deleted once the request is over.
+
+  Related behaviors:
+
+  * `Pleroma.Uploaders.Uploader`
+  * `Pleroma.Upload.Filter`
+
+  """
   alias Ecto.UUID
+  require Logger
 
-  @storage_backend Application.get_env(:pleroma, Pleroma.Upload)
-                   |> Keyword.fetch!(:uploader)
+  @type source ::
+          Plug.Upload.t() | data_uri_string ::
+          String.t() | {:from_local, name :: String.t(), id :: String.t(), path :: String.t()}
 
-  def store(%Plug.Upload{} = file, should_dedupe) do
-    content_type = get_content_type(file.path)
+  @type option ::
+          {:type, :avatar | :banner | :background}
+          | {:description, String.t()}
+          | {:activity_type, String.t()}
+          | {:size_limit, nil | non_neg_integer()}
+          | {:uploader, module()}
+          | {:filters, [module()]}
 
-    uuid = get_uuid(file, should_dedupe)
-    name = get_name(file, uuid, content_type, should_dedupe)
-
-    strip_exif_data(content_type, file.path)
-
-    {:ok, url_path} =
-      @storage_backend.put_file(name, uuid, file.path, content_type, should_dedupe)
-
-    %{
-      "type" => "Document",
-      "url" => [
-        %{
-          "type" => "Link",
-          "mediaType" => content_type,
-          "href" => url_path
+  @type t :: %__MODULE__{
+          id: String.t(),
+          name: String.t(),
+          tempfile: String.t(),
+          content_type: String.t(),
+          path: String.t()
         }
-      ],
-      "name" => name
-    }
+  defstruct [:id, :name, :tempfile, :content_type, :path]
+
+  @spec store(source, options :: [option()]) :: {:ok, Map.t()} | {:error, any()}
+  def store(upload, opts \\ []) do
+    opts = get_opts(opts)
+
+    with {:ok, upload} <- prepare_upload(upload, opts),
+         upload = %__MODULE__{upload | path: upload.path || "#{upload.id}/#{upload.name}"},
+         {:ok, upload} <- Pleroma.Upload.Filter.filter(opts.filters, upload),
+         {:ok, url_spec} <- Pleroma.Uploaders.Uploader.put_file(opts.uploader, upload) do
+      {:ok,
+       %{
+         "type" => opts.activity_type,
+         "url" => [
+           %{
+             "type" => "Link",
+             "mediaType" => upload.content_type,
+             "href" => url_from_spec(opts.base_url, url_spec)
+           }
+         ],
+         "name" => Map.get(opts, :description) || upload.name
+       }}
+    else
+      {:error, error} ->
+        Logger.error(
+          "#{__MODULE__} store (using #{inspect(opts.uploader)}) failed: #{inspect(error)}"
+        )
+
+        {:error, error}
+    end
   end
 
-  def store(%{"img" => "data:image/" <> image_data}, should_dedupe) do
+  defp get_opts(opts) do
+    {size_limit, activity_type} =
+      case Keyword.get(opts, :type) do
+        :banner ->
+          {Pleroma.Config.get!([:instance, :banner_upload_limit]), "Image"}
+
+        :avatar ->
+          {Pleroma.Config.get!([:instance, :avatar_upload_limit]), "Image"}
+
+        :background ->
+          {Pleroma.Config.get!([:instance, :background_upload_limit]), "Image"}
+
+        _ ->
+          {Pleroma.Config.get!([:instance, :upload_limit]), "Document"}
+      end
+
+    opts = %{
+      activity_type: Keyword.get(opts, :activity_type, activity_type),
+      size_limit: Keyword.get(opts, :size_limit, size_limit),
+      uploader: Keyword.get(opts, :uploader, Pleroma.Config.get([__MODULE__, :uploader])),
+      filters: Keyword.get(opts, :filters, Pleroma.Config.get([__MODULE__, :filters])),
+      description: Keyword.get(opts, :description),
+      base_url:
+        Keyword.get(
+          opts,
+          :base_url,
+          Pleroma.Config.get([__MODULE__, :base_url], Pleroma.Web.base_url())
+        )
+    }
+
+    # TODO: 1.0+ : remove old config compatibility
+    opts =
+      if Pleroma.Config.get([__MODULE__, :strip_exif]) == true &&
+           !Enum.member?(opts.filters, Pleroma.Upload.Filter.Mogrify) do
+        Logger.warn("""
+        Pleroma: configuration `:instance, :strip_exif` is deprecated, please instead set:
+
+          :pleroma, Pleroma.Upload, [filters: [Pleroma.Upload.Filter.Mogrify]]
+
+          :pleroma, Pleroma.Upload.Filter.Mogrify, args: "strip"
+        """)
+
+        Pleroma.Config.put([Pleroma.Upload.Filter.Mogrify], args: "strip")
+        Map.put(opts, :filters, opts.filters ++ [Pleroma.Upload.Filter.Mogrify])
+      else
+        opts
+      end
+
+    if Pleroma.Config.get([:instance, :dedupe_media]) == true &&
+         !Enum.member?(opts.filters, Pleroma.Upload.Filter.Dedupe) do
+      Logger.warn("""
+      Pleroma: configuration `:instance, :dedupe_media` is deprecated, please instead set:
+
+      :pleroma, Pleroma.Upload, [filters: [Pleroma.Upload.Filter.Dedupe]]
+      """)
+
+      Map.put(opts, :filters, opts.filters ++ [Pleroma.Upload.Filter.Dedupe])
+    else
+      opts
+    end
+  end
+
+  defp prepare_upload(%Plug.Upload{} = file, opts) do
+    with :ok <- check_file_size(file.path, opts.size_limit),
+         {:ok, content_type, name} <- Pleroma.MIME.file_mime_type(file.path, file.filename) do
+      {:ok,
+       %__MODULE__{
+         id: UUID.generate(),
+         name: name,
+         tempfile: file.path,
+         content_type: content_type
+       }}
+    end
+  end
+
+  defp prepare_upload(%{"img" => "data:image/" <> image_data}, opts) do
     parsed = Regex.named_captures(~r/(?<filetype>jpeg|png|gif);base64,(?<data>.*)/, image_data)
     data = Base.decode64!(parsed["data"], ignore: :whitespace)
+    hash = String.downcase(Base.encode16(:crypto.hash(:sha256, data)))
 
-    tmp_path = tempfile_for_image(data)
-
-    uuid = UUID.generate()
-
-    content_type = get_content_type(tmp_path)
-    strip_exif_data(content_type, tmp_path)
-
-    name =
-      create_name(
-        String.downcase(Base.encode16(:crypto.hash(:sha256, data))),
-        parsed["filetype"],
-        content_type
-      )
-
-    {:ok, url_path} = @storage_backend.put_file(name, uuid, tmp_path, content_type, should_dedupe)
-
-    %{
-      "type" => "Image",
-      "url" => [
-        %{
-          "type" => "Link",
-          "mediaType" => content_type,
-          "href" => url_path
-        }
-      ],
-      "name" => name
-    }
+    with :ok <- check_binary_size(data, opts.size_limit),
+         tmp_path <- tempfile_for_image(data),
+         {:ok, content_type, name} <-
+           Pleroma.MIME.bin_mime_type(data, hash <> "." <> parsed["filetype"]) do
+      {:ok,
+       %__MODULE__{
+         id: UUID.generate(),
+         name: name,
+         tempfile: tmp_path,
+         content_type: content_type
+       }}
+    end
   end
 
-  @doc """
-  Creates a tempfile using the Plug.Upload Genserver which cleans them up 
-  automatically.
-  """
-  def tempfile_for_image(data) do
+  # For Mix.Tasks.MigrateLocalUploads
+  defp prepare_upload(upload = %__MODULE__{tempfile: path}, _opts) do
+    with {:ok, content_type} <- Pleroma.MIME.file_mime_type(path) do
+      {:ok, %__MODULE__{upload | content_type: content_type}}
+    end
+  end
+
+  defp check_binary_size(binary, size_limit)
+       when is_integer(size_limit) and size_limit > 0 and byte_size(binary) >= size_limit do
+    {:error, :file_too_large}
+  end
+
+  defp check_binary_size(_, _), do: :ok
+
+  defp check_file_size(path, size_limit) when is_integer(size_limit) and size_limit > 0 do
+    with {:ok, %{size: size}} <- File.stat(path),
+         true <- size <= size_limit do
+      :ok
+    else
+      false -> {:error, :file_too_large}
+      error -> error
+    end
+  end
+
+  defp check_file_size(_, _), do: :ok
+
+  # Creates a tempfile using the Plug.Upload Genserver which cleans them up
+  # automatically.
+  defp tempfile_for_image(data) do
     {:ok, tmp_path} = Plug.Upload.random_file("profile_pics")
     {:ok, tmp_file} = File.open(tmp_path, [:write, :raw, :binary])
     IO.binwrite(tmp_file, data)
@@ -73,98 +210,10 @@ defmodule Pleroma.Upload do
     tmp_path
   end
 
-  def strip_exif_data(content_type, file) do
-    settings = Application.get_env(:pleroma, Pleroma.Upload)
-    do_strip = Keyword.fetch!(settings, :strip_exif)
-    [filetype, _ext] = String.split(content_type, "/")
-
-    if filetype == "image" and do_strip == true do
-      Mogrify.open(file) |> Mogrify.custom("strip") |> Mogrify.save(in_place: true)
-    end
+  defp url_from_spec(base_url, {:file, path}) do
+    [base_url, "media", path]
+    |> Path.join()
   end
 
-  defp create_name(uuid, ext, type) do
-    case type do
-      "application/octet-stream" ->
-        String.downcase(Enum.join([uuid, ext], "."))
-
-      "audio/mpeg" ->
-        String.downcase(Enum.join([uuid, "mp3"], "."))
-
-      _ ->
-        String.downcase(Enum.join([uuid, List.last(String.split(type, "/"))], "."))
-    end
-  end
-
-  defp get_uuid(file, should_dedupe) do
-    if should_dedupe do
-      Base.encode16(:crypto.hash(:sha256, File.read!(file.path)))
-    else
-      UUID.generate()
-    end
-  end
-
-  defp get_name(file, uuid, type, should_dedupe) do
-    if should_dedupe do
-      create_name(uuid, List.last(String.split(file.filename, ".")), type)
-    else
-      parts = String.split(file.filename, ".")
-
-      new_filename =
-        if length(parts) > 1 do
-          Enum.drop(parts, -1) |> Enum.join(".")
-        else
-          Enum.join(parts)
-        end
-
-      case type do
-        "application/octet-stream" -> file.filename
-        "audio/mpeg" -> new_filename <> ".mp3"
-        "image/jpeg" -> new_filename <> ".jpg"
-        _ -> Enum.join([new_filename, String.split(type, "/") |> List.last()], ".")
-      end
-    end
-  end
-
-  def get_content_type(file) do
-    match =
-      File.open(file, [:read], fn f ->
-        case IO.binread(f, 8) do
-          <<0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A>> ->
-            "image/png"
-
-          <<0x47, 0x49, 0x46, 0x38, _, 0x61, _, _>> ->
-            "image/gif"
-
-          <<0xFF, 0xD8, 0xFF, _, _, _, _, _>> ->
-            "image/jpeg"
-
-          <<0x1A, 0x45, 0xDF, 0xA3, _, _, _, _>> ->
-            "video/webm"
-
-          <<0x00, 0x00, 0x00, _, 0x66, 0x74, 0x79, 0x70>> ->
-            "video/mp4"
-
-          <<0x49, 0x44, 0x33, _, _, _, _, _>> ->
-            "audio/mpeg"
-
-          <<255, 251, _, 68, 0, 0, 0, 0>> ->
-            "audio/mpeg"
-
-          <<0x4F, 0x67, 0x67, 0x53, 0x00, 0x02, 0x00, 0x00>> ->
-            "audio/ogg"
-
-          <<0x52, 0x49, 0x46, 0x46, _, _, _, _>> ->
-            "audio/wav"
-
-          _ ->
-            "application/octet-stream"
-        end
-      end)
-
-    case match do
-      {:ok, type} -> type
-      _e -> "application/octet-stream"
-    end
-  end
+  defp url_from_spec(_base_url, {:url, url}), do: url
 end

lib/pleroma/upload/filter.ex

@@ -0,0 +1,35 @@
+defmodule Pleroma.Upload.Filter do
+  @moduledoc """
+  Upload Filter behaviour
+
+  This behaviour allows to run filtering actions just before a file is uploaded. This allows to:
+
+  * morph in place the temporary file
+  * change any field of a `Pleroma.Upload` struct
+  * cancel/stop the upload
+  """
+
+  require Logger
+
+  @callback filter(Pleroma.Upload.t()) :: :ok | {:ok, Pleroma.Upload.t()} | {:error, any()}
+
+  @spec filter([module()], Pleroma.Upload.t()) :: {:ok, Pleroma.Upload.t()} | {:error, any()}
+
+  def filter([], upload) do
+    {:ok, upload}
+  end
+
+  def filter([filter | rest], upload) do
+    case filter.filter(upload) do
+      :ok ->
+        filter(rest, upload)
+
+      {:ok, upload} ->
+        filter(rest, upload)
+
+      error ->
+        Logger.error("#{__MODULE__}: Filter #{filter} failed: #{inspect(error)}")
+        error
+    end
+  end
+end

lib/pleroma/upload/filter/anonymize_filename.ex

@@ -0,0 +1,23 @@
+defmodule Pleroma.Upload.Filter.AnonymizeFilename do
+  @moduledoc """
+  Replaces the original filename with a pre-defined text or randomly generated string.
+
+  Should be used after `Pleroma.Upload.Filter.Dedupe`.
+  """
+  @behaviour Pleroma.Upload.Filter
+
+  def filter(upload) do
+    extension = List.last(String.split(upload.name, "."))
+    name = Pleroma.Config.get([__MODULE__, :text], random(extension))
+    {:ok, %Pleroma.Upload{upload | name: name}}
+  end
+
+  defp random(extension) do
+    string =
+      10
+      |> :crypto.strong_rand_bytes()
+      |> Base.url_encode64(padding: false)
+
+    string <> "." <> extension
+  end
+end

lib/pleroma/upload/filter/dedupe.ex

@@ -0,0 +1,11 @@
+defmodule Pleroma.Upload.Filter.Dedupe do
+  @behaviour Pleroma.Upload.Filter
+  alias Pleroma.Upload
+
+  def filter(upload = %Upload{name: name}) do
+    extension = String.split(name, ".") |> List.last()
+    shasum = :crypto.hash(:sha256, File.read!(upload.tempfile)) |> Base.encode16(case: :lower)
+    filename = shasum <> "." <> extension
+    {:ok, %Upload{upload | id: shasum, path: filename}}
+  end
+end

lib/pleroma/upload/filter/mogrifun.ex

@@ -0,0 +1,60 @@
+defmodule Pleroma.Upload.Filter.Mogrifun do
+  @behaviour Pleroma.Upload.Filter
+
+  @filters [
+    {"implode", "1"},
+    {"-raise", "20"},
+    {"+raise", "20"},
+    [{"-interpolate", "nearest"}, {"-virtual-pixel", "mirror"}, {"-spread", "5"}],
+    "+polaroid",
+    {"-statistic", "Mode 10"},
+    {"-emboss", "0x1.1"},
+    {"-emboss", "0x2"},
+    {"-colorspace", "Gray"},
+    "-negate",
+    [{"-channel", "green"}, "-negate"],
+    [{"-channel", "red"}, "-negate"],
+    [{"-channel", "blue"}, "-negate"],
+    {"+level-colors", "green,gold"},
+    {"+level-colors", ",DodgerBlue"},
+    {"+level-colors", ",Gold"},
+    {"+level-colors", ",Lime"},
+    {"+level-colors", ",Red"},
+    {"+level-colors", ",DarkGreen"},
+    {"+level-colors", "firebrick,yellow"},
+    {"+level-colors", "'rgb(102,75,25)',lemonchiffon"},
+    [{"fill", "red"}, {"tint", "40"}],
+    [{"fill", "green"}, {"tint", "40"}],
+    [{"fill", "blue"}, {"tint", "40"}],
+    [{"fill", "yellow"}, {"tint", "40"}]
+  ]
+
+  def filter(%Pleroma.Upload{tempfile: file, content_type: "image" <> _}) do
+    filter = Enum.random(@filters)
+
+    file
+    |> Mogrify.open()
+    |> mogrify_filter(filter)
+    |> Mogrify.save(in_place: true)
+
+    :ok
+  end
+
+  def filter(_), do: :ok
+
+  defp mogrify_filter(mogrify, [filter | rest]) do
+    mogrify
+    |> mogrify_filter(filter)
+    |> mogrify_filter(rest)
+  end
+
+  defp mogrify_filter(mogrify, []), do: mogrify
+
+  defp mogrify_filter(mogrify, {action, options}) do
+    Mogrify.custom(mogrify, action, options)
+  end
+
+  defp mogrify_filter(mogrify, string) when is_binary(string) do
+    Mogrify.custom(mogrify, string)
+  end
+end

lib/pleroma/upload/filter/mogrify.ex

@@ -0,0 +1,37 @@
+defmodule Pleroma.Upload.Filter.Mogrify do
+  @behaviour Pleroma.Upload.Filter
+
+  @type conversion :: action :: String.t() | {action :: String.t(), opts :: String.t()}
+  @type conversions :: conversion() | [conversion()]
+
+  def filter(%Pleroma.Upload{tempfile: file, content_type: "image" <> _}) do
+    filters = Pleroma.Config.get!([__MODULE__, :args])
+
+    file
+    |> Mogrify.open()
+    |> mogrify_filter(filters)
+    |> Mogrify.save(in_place: true)
+
+    :ok
+  end
+
+  def filter(_), do: :ok
+
+  defp mogrify_filter(mogrify, nil), do: mogrify
+
+  defp mogrify_filter(mogrify, [filter | rest]) do
+    mogrify
+    |> mogrify_filter(filter)
+    |> mogrify_filter(rest)
+  end
+
+  defp mogrify_filter(mogrify, []), do: mogrify
+
+  defp mogrify_filter(mogrify, {action, options}) do
+    Mogrify.custom(mogrify, action, options)
+  end
+
+  defp mogrify_filter(mogrify, action) when is_binary(action) do
+    Mogrify.custom(mogrify, action)
+  end
+end

lib/pleroma/uploaders/local.ex

@@ -1,51 +1,32 @@
 defmodule Pleroma.Uploaders.Local do
   @behaviour Pleroma.Uploaders.Uploader
 
-  alias Pleroma.Web
+  def get_file(_) do
+    {:ok, {:static_dir, upload_path()}}
+  end
 
-  def put_file(name, uuid, tmpfile, _content_type, should_dedupe) do
-    upload_folder = get_upload_path(uuid, should_dedupe)
-    url_path = get_url(name, uuid, should_dedupe)
+  def put_file(upload) do
+    {local_path, file} =
+      case Enum.reverse(String.split(upload.path, "/", trim: true)) do
+        [file] ->
+          {upload_path(), file}
 
-    File.mkdir_p!(upload_folder)
+        [file | folders] ->
+          path = Path.join([upload_path()] ++ Enum.reverse(folders))
+          File.mkdir_p!(path)
+          {path, file}
+      end
 
-    result_file = Path.join(upload_folder, name)
+    result_file = Path.join(local_path, file)
 
-    if File.exists?(result_file) do
-      File.rm!(tmpfile)
-    else
-      File.cp!(tmpfile, result_file)
+    unless File.exists?(result_file) do
+      File.cp!(upload.tempfile, result_file)
     end
 
-    {:ok, url_path}
+    :ok
   end
 
   def upload_path do
-    settings = Application.get_env(:pleroma, Pleroma.Uploaders.Local)
-    Keyword.fetch!(settings, :uploads)
-  end
-
-  defp get_upload_path(uuid, should_dedupe) do
-    if should_dedupe do
-      upload_path()
-    else
-      Path.join(upload_path(), uuid)
-    end
-  end
-
-  defp get_url(name, uuid, should_dedupe) do
-    if should_dedupe do
-      url_for(:cow_uri.urlencode(name))
-    else
-      url_for(Path.join(uuid, :cow_uri.urlencode(name)))
-    end
-  end
-
-  defp url_for(file) do
-    settings = Application.get_env(:pleroma, Pleroma.Uploaders.Local)
-
-    Keyword.get(settings, :uploads_url)
-    |> String.replace("{{file}}", file)
-    |> String.replace("{{base_url}}", Web.base_url())
+    Pleroma.Config.get!([__MODULE__, :uploads])
   end
 end

lib/pleroma/uploaders/mdii.ex

@@ -0,0 +1,31 @@
+defmodule Pleroma.Uploaders.MDII do
+  alias Pleroma.Config
+
+  @behaviour Pleroma.Uploaders.Uploader
+
+  @httpoison Application.get_env(:pleroma, :httpoison)
+
+  # MDII-hosted images are never passed through the MediaPlug; only local media.
+  # Delegate to Pleroma.Uploaders.Local
+  def get_file(file) do
+    Pleroma.Uploaders.Local.get_file(file)
+  end
+
+  def put_file(upload) do
+    cgi = Config.get([Pleroma.Uploaders.MDII, :cgi])
+    files = Config.get([Pleroma.Uploaders.MDII, :files])
+
+    {:ok, file_data} = File.read(upload.tempfile)
+
+    extension = String.split(upload.name, ".") |> List.last()
+    query = "#{cgi}?#{extension}"
+
+    with {:ok, %{status: 200, body: body}} <- @httpoison.post(query, file_data) do
+      remote_file_name = String.split(body) |> List.first()
+      public_url = "#{files}/#{remote_file_name}.#{extension}"
+      {:ok, {:url, public_url}}
+    else
+      _ -> Pleroma.Uploaders.Local.put_file(upload)
+    end
+  end
+end

lib/pleroma/uploaders/s3.ex

@@ -1,24 +1,46 @@
 defmodule Pleroma.Uploaders.S3 do
   @behaviour Pleroma.Uploaders.Uploader
+  require Logger
 
-  def put_file(name, uuid, path, content_type, _should_dedupe) do
-    settings = Application.get_env(:pleroma, Pleroma.Uploaders.S3)
-    bucket = Keyword.fetch!(settings, :bucket)
-    public_endpoint = Keyword.fetch!(settings, :public_endpoint)
+  # The file name is re-encoded with S3's constraints here to comply with previous links with less strict filenames
+  def get_file(file) do
+    config = Pleroma.Config.get([__MODULE__])
 
-    {:ok, file_data} = File.read(path)
+    {:ok,
+     {:url,
+      Path.join([
+        Keyword.fetch!(config, :public_endpoint),
+        Keyword.fetch!(config, :bucket),
+        strict_encode(URI.decode(file))
+      ])}}
+  end
 
-    File.rm!(path)
+  def put_file(upload = %Pleroma.Upload{}) do
+    config = Pleroma.Config.get([__MODULE__])
+    bucket = Keyword.get(config, :bucket)
 
-    s3_name = "#{uuid}/#{name}"
+    {:ok, file_data} = File.read(upload.tempfile)
 
-    {:ok, _} =
+    s3_name = strict_encode(upload.path)
+
+    op =
       ExAws.S3.put_object(bucket, s3_name, file_data, [
         {:acl, :public_read},
-        {:content_type, content_type}
+        {:content_type, upload.content_type}
       ])
-      |> ExAws.request()
 
-    {:ok, "#{public_endpoint}/#{bucket}/#{s3_name}"}
+    case ExAws.request(op) do
+      {:ok, _} ->
+        {:ok, {:file, s3_name}}
+
+      error ->
+        Logger.error("#{__MODULE__}: #{inspect(error)}")
+        {:error, "S3 Upload failed"}
+    end
+  end
+
+  @regex Regex.compile!("[^0-9a-zA-Z!.*/'()_-]")
+  def strict_encode(name) do
+    String.replace(name, @regex, "-")
   end
 end

lib/pleroma/uploaders/swift/keystone.ex

@@ -1,11 +1,9 @@
 defmodule Pleroma.Uploaders.Swift.Keystone do
   use HTTPoison.Base
 
-  @settings Application.get_env(:pleroma, Pleroma.Uploaders.Swift)
-
   def process_url(url) do
     Enum.join(
-      [Keyword.fetch!(@settings, :auth_url), url],
+      [Pleroma.Config.get!([Pleroma.Uploaders.Swift, :auth_url]), url],
       "/"
     )
   end
@@ -16,9 +14,10 @@ defmodule Pleroma.Uploaders.Swift.Keystone do
   end
 
   def get_token() do
-    username = Keyword.fetch!(@settings, :username)
-    password = Keyword.fetch!(@settings, :password)
-    tenant_id = Keyword.fetch!(@settings, :tenant_id)
+    settings = Pleroma.Config.get(Pleroma.Uploaders.Swift)
+    username = Keyword.fetch!(settings, :username)
+    password = Keyword.fetch!(settings, :password)
+    tenant_id = Keyword.fetch!(settings, :tenant_id)
 
     case post(
            "/tokens",
@@ -26,10 +25,10 @@ defmodule Pleroma.Uploaders.Swift.Keystone do
            ["Content-Type": "application/json"],
            hackney: [:insecure]
          ) do
-      {:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
+      {:ok, %Tesla.Env{status: 200, body: body}} ->
         body["access"]["token"]["id"]
 
-      {:ok, %HTTPoison.Response{status_code: _}} ->
+      {:ok, %Tesla.Env{status: _}} ->
         ""
     end
   end

lib/pleroma/uploaders/swift/swift.ex

@@ -1,24 +1,21 @@
 defmodule Pleroma.Uploaders.Swift.Client do
   use HTTPoison.Base
 
-  @settings Application.get_env(:pleroma, Pleroma.Uploaders.Swift)
-
   def process_url(url) do
     Enum.join(
-      [Keyword.fetch!(@settings, :storage_url), url],
+      [Pleroma.Config.get!([Pleroma.Uploaders.Swift, :storage_url]), url],
       "/"
     )
   end
 
   def upload_file(filename, body, content_type) do
-    object_url = Keyword.fetch!(@settings, :object_url)
     token = Pleroma.Uploaders.Swift.Keystone.get_token()
 
     case put("#{filename}", body, "X-Auth-Token": token, "Content-Type": content_type) do
-      {:ok, %HTTPoison.Response{status_code: 201}} ->
-        {:ok, "#{object_url}/#{filename}"}
+      {:ok, %Tesla.Env{status: 201}} ->
+        {:ok, {:file, filename}}
 
-      {:ok, %HTTPoison.Response{status_code: 401}} ->
+      {:ok, %Tesla.Env{status: 401}} ->
         {:error, "Unauthorized, Bad Token"}
 
       {:error, _} ->

lib/pleroma/uploaders/swift/uploader.ex

@@ -1,10 +1,15 @@
 defmodule Pleroma.Uploaders.Swift do
   @behaviour Pleroma.Uploaders.Uploader
 
-  def put_file(name, uuid, tmp_path, content_type, _should_dedupe) do
-    {:ok, file_data} = File.read(tmp_path)
-    remote_name = "#{uuid}/#{name}"
+  def get_file(name) do
+    {:ok, {:url, Path.join([Pleroma.Config.get!([__MODULE__, :object_url]), name])}}
+  end
 
-    Pleroma.Uploaders.Swift.Client.upload_file(remote_name, file_data, content_type)
+  def put_file(upload) do
+    Pleroma.Uploaders.Swift.Client.upload_file(
+      upload.path,
+      File.read!(upload.tmpfile),
+      upload.content_type
+    )
   end
 end

lib/pleroma/uploaders/uploader.ex

@@ -1,20 +1,40 @@
 defmodule Pleroma.Uploaders.Uploader do
   @moduledoc """
-  Defines the contract to put an uploaded file to any backend.
+  Defines the contract to put and get an uploaded file to any backend.
   """
 
+  @doc """
+  Instructs how to get the file from the backend.
+
+  Used by `Pleroma.Plugs.UploadedMedia`.
+  """
+  @type get_method :: {:static_dir, directory :: String.t()} | {:url, url :: String.t()}
+  @callback get_file(file :: String.t()) :: {:ok, get_method()}
+
   @doc """
   Put a file to the backend.
 
-  Returns `{:ok, String.t } | {:error, String.t} containing the path of the 
-  uploaded file, or error information if the file failed to be saved to the 
-  respective backend.
+  Returns:
+
+  * `:ok` which assumes `{:ok, upload.path}`
+  * `{:ok, spec}` where spec is:
+    * `{:file, filename :: String.t}` to handle reads with `get_file/1` (recommended)
+
+    This allows to correctly proxy or redirect requests to the backend, while allowing to migrate backends without breaking any URL.
+  * `{url, url :: String.t}` to bypass `get_file/2` and use the `url` directly in the activity.
+  * `{:error, String.t}` error information if the file failed to be saved to the backend.
+
+
   """
-  @callback put_file(
-              name :: String.t(),
-              uuid :: String.t(),
-              file :: File.t(),
-              content_type :: String.t(),
-              should_dedupe :: Boolean.t()
-            ) :: {:ok, String.t()} | {:error, String.t()}
+  @callback put_file(Pleroma.Upload.t()) ::
+              :ok | {:ok, {:file | :url, String.t()}} | {:error, String.t()}
+
+  @spec put_file(module(), Pleroma.Upload.t()) ::
+          {:ok, {:file | :url, String.t()}} | {:error, String.t()}
+  def put_file(uploader, upload) do
+    case uploader.put_file(upload) do
+      :ok -> {:ok, {:file, upload.path}}
+      other -> other
+    end
+  end
 end

lib/pleroma/user.ex

@@ -4,9 +4,13 @@ defmodule Pleroma.User do
   import Ecto.{Changeset, Query}
   alias Pleroma.{Repo, User, Object, Web, Activity, Notification}
   alias Comeonin.Pbkdf2
-  alias Pleroma.Web.{OStatus, Websub}
+  alias Pleroma.Formatter
+  alias Pleroma.Web.CommonAPI.Utils, as: CommonUtils
+  alias Pleroma.Web.{OStatus, Websub, OAuth}
   alias Pleroma.Web.ActivityPub.{Utils, ActivityPub}
 
+  @type t :: %__MODULE__{}
+
   schema "users" do
     field(:bio, :string)
     field(:email, :string)
@@ -19,10 +23,12 @@ defmodule Pleroma.User do
     field(:ap_id, :string)
     field(:avatar, :map)
     field(:local, :boolean, default: true)
-    field(:info, :map, default: %{})
     field(:follower_address, :string)
     field(:search_distance, :float, virtual: true)
+    field(:tags, {:array, :string}, default: [])
+    field(:last_refreshed_at, :naive_datetime)
     has_many(:notifications, Notification)
+    embeds_one(:info, Pleroma.User.Info)
 
     timestamps()
   end
@@ -35,12 +41,16 @@ defmodule Pleroma.User do
   end
 
   def banner_url(user) do
-    case user.info["banner"] do
+    case user.info.banner do
       %{"url" => [%{"href" => href} | _]} -> href
       _ -> "#{Web.base_url()}/images/banner.png"
     end
   end
 
+  def profile_url(%User{info: %{source_data: %{"url" => url}}}), do: url
+  def profile_url(%User{ap_id: ap_id}), do: ap_id
+  def profile_url(_), do: nil
+
   def ap_id(%User{nickname: nickname}) do
     "#{Web.base_url()}/users/#{nickname}"
   end
@@ -55,38 +65,39 @@ defmodule Pleroma.User do
     |> validate_required([:following])
   end
 
-  def info_changeset(struct, params \\ %{}) do
-    struct
-    |> cast(params, [:info])
-    |> validate_required([:info])
-  end
-
   def user_info(%User{} = user) do
     oneself = if user.local, do: 1, else: 0
 
     %{
       following_count: length(user.following) - oneself,
-      note_count: user.info["note_count"] || 0,
-      follower_count: user.info["follower_count"] || 0,
-      locked: user.info["locked"] || false,
-      default_scope: user.info["default_scope"] || "public"
+      note_count: user.info.note_count,
+      follower_count: user.info.follower_count,
+      locked: user.info.locked,
+      default_scope: user.info.default_scope
     }
   end
 
   @email_regex ~r/^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/
   def remote_user_creation(params) do
+    params =
+      params
+      |> Map.put(:info, params[:info] || %{})
+
+    info_cng = User.Info.remote_user_creation(%User.Info{}, params[:info])
+
     changes =
       %User{}
-      |> cast(params, [:bio, :name, :ap_id, :nickname, :info, :avatar])
+      |> cast(params, [:bio, :name, :ap_id, :nickname, :avatar])
       |> validate_required([:name, :ap_id])
       |> unique_constraint(:nickname)
       |> validate_format(:nickname, @email_regex)
       |> validate_length(:bio, max: 5000)
       |> validate_length(:name, max: 100)
       |> put_change(:local, false)
+      |> put_embed(:info, info_cng)
 
     if changes.valid? do
-      case changes.changes[:info]["source_data"] do
+      case info_cng.changes[:source_data] do
         %{"followers" => followers} ->
           changes
           |> put_change(:follower_address, followers)
@@ -104,7 +115,7 @@ defmodule Pleroma.User do
 
   def update_changeset(struct, params \\ %{}) do
     struct
-    |> cast(params, [:bio, :name])
+    |> cast(params, [:bio, :name, :avatar])
     |> unique_constraint(:nickname)
     |> validate_format(:nickname, ~r/^[a-zA-Z\d]+$/)
     |> validate_length(:bio, max: 5000)
@@ -112,12 +123,21 @@ defmodule Pleroma.User do
   end
 
   def upgrade_changeset(struct, params \\ %{}) do
+    params =
+      params
+      |> Map.put(:last_refreshed_at, NaiveDateTime.utc_now())
+
+    info_cng =
+      struct.info
+      |> User.Info.user_upgrade(params[:info])
+
     struct
-    |> cast(params, [:bio, :name, :info, :follower_address, :avatar])
+    |> cast(params, [:bio, :name, :follower_address, :avatar, :last_refreshed_at])
     |> unique_constraint(:nickname)
     |> validate_format(:nickname, ~r/^[a-zA-Z\d]+$/)
     |> validate_length(:bio, max: 5000)
     |> validate_length(:name, max: 100)
+    |> put_embed(:info, info_cng)
   end
 
   def password_update_changeset(struct, params) do
@@ -127,6 +147,9 @@ defmodule Pleroma.User do
       |> validate_required([:password, :password_confirmation])
       |> validate_confirmation(:password)
 
+    OAuth.Token.delete_user_tokens(struct)
+    OAuth.Authorization.delete_user_authorizations(struct)
+
     if changeset.valid? do
       hashed = Pbkdf2.hashpwsalt(changeset.changes[:password])
 
@@ -153,6 +176,7 @@ defmodule Pleroma.User do
       |> validate_format(:email, @email_regex)
       |> validate_length(:bio, max: 1000)
       |> validate_length(:name, min: 1, max: 100)
+      |> put_change(:info, %Pleroma.User.Info{})
 
     if changeset.valid? do
       hashed = Pbkdf2.hashpwsalt(changeset.changes[:password])
@@ -169,40 +193,33 @@ defmodule Pleroma.User do
     end
   end
 
-  def maybe_direct_follow(%User{} = follower, %User{info: info} = followed) do
-    user_config = Application.get_env(:pleroma, :user)
-    deny_follow_blocked = Keyword.get(user_config, :deny_follow_blocked)
+  def needs_update?(%User{local: true}), do: false
 
-    user_info = user_info(followed)
+  def needs_update?(%User{local: false, last_refreshed_at: nil}), do: true
 
-    should_direct_follow =
-      cond do
-        # if the account is locked, don't pre-create the relationship
-        user_info[:locked] == true ->
-          false
+  def needs_update?(%User{local: false} = user) do
+    NaiveDateTime.diff(NaiveDateTime.utc_now(), user.last_refreshed_at) >= 86400
+  end
 
-        # if the users are blocking each other, we shouldn't even be here, but check for it anyway
-        deny_follow_blocked and
-            (User.blocks?(follower, followed) or User.blocks?(followed, follower)) ->
-          false
+  def needs_update?(_), do: true
 
-        # if OStatus, then there is no three-way handshake to follow
-        User.ap_enabled?(followed) != true ->
-          true
+  def maybe_direct_follow(%User{} = follower, %User{local: true, info: %{locked: true}}) do
+    {:ok, follower}
+  end
 
-        # if there are no other reasons not to, just pre-create the relationship
-        true ->
-          true
-      end
+  def maybe_direct_follow(%User{} = follower, %User{local: true} = followed) do
+    follow(follower, followed)
+  end
 
-    if should_direct_follow do
+  def maybe_direct_follow(%User{} = follower, %User{} = followed) do
+    if !User.ap_enabled?(followed) do
       follow(follower, followed)
     else
       {:ok, follower}
     end
   end
 
-  def maybe_follow(%User{} = follower, %User{info: info} = followed) do
+  def maybe_follow(%User{} = follower, %User{info: _info} = followed) do
     if not following?(follower, followed) do
       follow(follower, followed)
     else
@@ -217,7 +234,7 @@ defmodule Pleroma.User do
     ap_followers = followed.follower_address
 
     cond do
-      following?(follower, followed) or info["deactivated"] ->
+      following?(follower, followed) or info.deactivated ->
         {:error, "Could not follow user: #{followed.nickname} is already on your list."}
 
       deny_follow_blocked and blocks?(followed, follower) ->
@@ -264,12 +281,13 @@ defmodule Pleroma.User do
     end
   end
 
+  @spec following?(User.t(), User.t()) :: boolean
   def following?(%User{} = follower, %User{} = followed) do
     Enum.member?(follower.following, followed.follower_address)
   end
 
   def locked?(%User{} = user) do
-    user.info["locked"] || false
+    user.info.locked || false
   end
 
   def get_by_ap_id(ap_id) do
@@ -290,6 +308,7 @@ defmodule Pleroma.User do
   def invalidate_cache(user) do
     Cachex.del(:user_cache, "ap_id:#{user.ap_id}")
     Cachex.del(:user_cache, "nickname:#{user.nickname}")
+    Cachex.del(:user_cache, "user_info:#{user.id}")
   end
 
   def get_cached_by_ap_id(ap_id) do
@@ -405,22 +424,23 @@ defmodule Pleroma.User do
   end
 
   def increase_note_count(%User{} = user) do
-    note_count = (user.info["note_count"] || 0) + 1
-    new_info = Map.put(user.info, "note_count", note_count)
+    info_cng = User.Info.add_to_note_count(user.info, 1)
 
-    cs = info_changeset(user, %{info: new_info})
+    cng =
+      change(user)
+      |> put_embed(:info, info_cng)
 
-    update_and_set_cache(cs)
+    update_and_set_cache(cng)
   end
 
   def decrease_note_count(%User{} = user) do
-    note_count = user.info["note_count"] || 0
-    note_count = if note_count <= 0, do: 0, else: note_count - 1
-    new_info = Map.put(user.info, "note_count", note_count)
+    info_cng = User.Info.add_to_note_count(user.info, -1)
 
-    cs = info_changeset(user, %{info: new_info})
+    cng =
+      change(user)
+      |> put_embed(:info, info_cng)
 
-    update_and_set_cache(cs)
+    update_and_set_cache(cng)
   end
 
   def update_note_count(%User{} = user) do
@@ -433,11 +453,13 @@ defmodule Pleroma.User do
 
     note_count = Repo.one(note_count_query)
 
-    new_info = Map.put(user.info, "note_count", note_count)
+    info_cng = User.Info.set_note_count(user.info, note_count)
 
-    cs = info_changeset(user, %{info: new_info})
+    cng =
+      change(user)
+      |> put_embed(:info, info_cng)
 
-    update_and_set_cache(cs)
+    update_and_set_cache(cng)
   end
 
   def update_follower_count(%User{} = user) do
@@ -451,43 +473,36 @@ defmodule Pleroma.User do
 
     follower_count = Repo.one(follower_count_query)
 
-    new_info = Map.put(user.info, "follower_count", follower_count)
+    info_cng =
+      user.info
+      |> User.Info.set_follower_count(follower_count)
 
-    cs = info_changeset(user, %{info: new_info})
+    cng =
+      change(user)
+      |> put_embed(:info, info_cng)
 
-    update_and_set_cache(cs)
+    update_and_set_cache(cng)
   end
 
-  def get_notified_from_activity_query(to) do
+  def get_users_from_set_query(ap_ids, false) do
     from(
       u in User,
-      where: u.ap_id in ^to,
+      where: u.ap_id in ^ap_ids
+    )
+  end
+
+  def get_users_from_set_query(ap_ids, true) do
+    query = get_users_from_set_query(ap_ids, false)
+
+    from(
+      u in query,
       where: u.local == true
     )
   end
 
-  def get_notified_from_activity(%Activity{recipients: to, data: %{"type" => "Announce"} = data}) do
-    object = Object.normalize(data["object"])
-    actor = User.get_cached_by_ap_id(data["actor"])
-
-    # ensure that the actor who published the announced object appears only once
-    to =
-      if actor.nickname != nil do
-        to ++ [object.data["actor"]]
-      else
-        to
-      end
-      |> Enum.uniq()
-
-    query = get_notified_from_activity_query(to)
-
-    Repo.all(query)
-  end
-
-  def get_notified_from_activity(%Activity{recipients: to}) do
-    query = get_notified_from_activity_query(to)
-
-    Repo.all(query)
+  def get_users_from_set(ap_ids, local_only \\ true) do
+    get_users_from_set_query(ap_ids, local_only)
+    |> Repo.all()
   end
 
   def get_recipients_from_activity(%Activity{recipients: to}) do
@@ -503,7 +518,7 @@ defmodule Pleroma.User do
     Repo.all(query)
   end
 
-  def search(query, resolve) do
+  def search(query, resolve \\ false) do
     # strip the beginning @ off if there is a query
     query = String.trim_leading(query, "@")
 
@@ -550,12 +565,15 @@ defmodule Pleroma.User do
       unfollow(blocked, blocker)
     end
 
-    blocks = blocker.info["blocks"] || []
-    new_blocks = Enum.uniq([ap_id | blocks])
-    new_info = Map.put(blocker.info, "blocks", new_blocks)
+    info_cng =
+      blocker.info
+      |> User.Info.add_to_block(ap_id)
 
-    cs = User.info_changeset(blocker, %{info: new_info})
-    update_and_set_cache(cs)
+    cng =
+      change(blocker)
+      |> put_embed(:info, info_cng)
+
+    update_and_set_cache(cng)
   end
 
   # helper to handle the block given only an actor's AP id
@@ -563,18 +581,21 @@ defmodule Pleroma.User do
     block(blocker, User.get_by_ap_id(ap_id))
   end
 
-  def unblock(user, %{ap_id: ap_id}) do
-    blocks = user.info["blocks"] || []
-    new_blocks = List.delete(blocks, ap_id)
-    new_info = Map.put(user.info, "blocks", new_blocks)
+  def unblock(blocker, %{ap_id: ap_id}) do
+    info_cng =
+      blocker.info
+      |> User.Info.remove_from_block(ap_id)
 
-    cs = User.info_changeset(user, %{info: new_info})
-    update_and_set_cache(cs)
+    cng =
+      change(blocker)
+      |> put_embed(:info, info_cng)
+
+    update_and_set_cache(cng)
   end
 
   def blocks?(user, %{ap_id: ap_id}) do
-    blocks = user.info["blocks"] || []
-    domain_blocks = user.info["domain_blocks"] || []
+    blocks = user.info.blocks
+    domain_blocks = user.info.domain_blocks
     %{host: host} = URI.parse(ap_id)
 
     Enum.member?(blocks, ap_id) ||
@@ -584,21 +605,27 @@ defmodule Pleroma.User do
   end
 
   def block_domain(user, domain) do
-    domain_blocks = user.info["domain_blocks"] || []
-    new_blocks = Enum.uniq([domain | domain_blocks])
-    new_info = Map.put(user.info, "domain_blocks", new_blocks)
+    info_cng =
+      user.info
+      |> User.Info.add_to_domain_block(domain)
 
-    cs = User.info_changeset(user, %{info: new_info})
-    update_and_set_cache(cs)
+    cng =
+      change(user)
+      |> put_embed(:info, info_cng)
+
+    update_and_set_cache(cng)
   end
 
   def unblock_domain(user, domain) do
-    blocks = user.info["domain_blocks"] || []
-    new_blocks = List.delete(blocks, domain)
-    new_info = Map.put(user.info, "domain_blocks", new_blocks)
+    info_cng =
+      user.info
+      |> User.Info.remove_from_domain_block(domain)
 
-    cs = User.info_changeset(user, %{info: new_info})
-    update_and_set_cache(cs)
+    cng =
+      change(user)
+      |> put_embed(:info, info_cng)
+
+    update_and_set_cache(cng)
   end
 
   def local_user_query() do
@@ -617,10 +644,14 @@ defmodule Pleroma.User do
     )
   end
 
-  def deactivate(%User{} = user) do
-    new_info = Map.put(user.info, "deactivated", true)
-    cs = User.info_changeset(user, %{info: new_info})
-    update_and_set_cache(cs)
+  def deactivate(%User{} = user, status \\ true) do
+    info_cng = User.Info.set_activation_status(user.info, status)
+
+    cng =
+      change(user)
+      |> put_embed(:info, info_cng)
+
+    update_and_set_cache(cng)
   end
 
   def delete(%User{} = user) do
@@ -651,11 +682,19 @@ defmodule Pleroma.User do
       end
     end)
 
-    :ok
+    {:ok, user}
   end
 
+  def html_filter_policy(%User{info: %{no_rich_text: true}}) do
+    Pleroma.HTML.Scrubber.TwitterText
+  end
+
+  def html_filter_policy(_), do: nil
+
   def get_or_fetch_by_ap_id(ap_id) do
-    if user = get_by_ap_id(ap_id) do
+    user = get_by_ap_id(ap_id)
+
+    if !is_nil(user) and !User.needs_update?(user) do
       user
     else
       ap_try = ActivityPub.make_user_from_ap_id(ap_id)
@@ -680,7 +719,7 @@ defmodule Pleroma.User do
       user
     else
       changes =
-        %User{}
+        %User{info: %User.Info{}}
         |> cast(%{}, [:ap_id, :nickname, :local])
         |> put_change(:ap_id, relay_uri)
         |> put_change(:nickname, nil)
@@ -694,7 +733,7 @@ defmodule Pleroma.User do
 
   # AP style
   def public_key_from_info(%{
-        "source_data" => %{"publicKey" => %{"publicKeyPem" => public_key_pem}}
+        source_data: %{"publicKey" => %{"publicKeyPem" => public_key_pem}}
       }) do
     key =
       :public_key.pem_decode(public_key_pem)
@@ -705,7 +744,7 @@ defmodule Pleroma.User do
   end
 
   # OStatus Magic Key
-  def public_key_from_info(%{"magic_key" => magic_key}) do
+  def public_key_from_info(%{magic_key: magic_key}) do
     {:ok, Pleroma.Web.Salmon.decode_key(magic_key)}
   end
 
@@ -727,10 +766,12 @@ defmodule Pleroma.User do
       |> Map.put(:name, blank?(data[:name]) || data[:nickname])
 
     cs = User.remote_user_creation(data)
+
     Repo.insert(cs, on_conflict: :replace_all, conflict_target: :nickname)
   end
 
-  def ap_enabled?(%User{info: info}), do: info["ap_enabled"]
+  def ap_enabled?(%User{local: true}), do: true
+  def ap_enabled?(%User{info: info}), do: info.ap_enabled
   def ap_enabled?(_), do: false
 
   def get_or_fetch(uri_or_nickname) do
@@ -740,4 +781,84 @@ defmodule Pleroma.User do
       get_or_fetch_by_nickname(uri_or_nickname)
     end
   end
+
+  # wait a period of time and return newest version of the User structs
+  # this is because we have synchronous follow APIs and need to simulate them
+  # with an async handshake
+  def wait_and_refresh(_, %User{local: true} = a, %User{local: true} = b) do
+    with %User{} = a <- Repo.get(User, a.id),
+         %User{} = b <- Repo.get(User, b.id) do
+      {:ok, a, b}
+    else
+      _e ->
+        :error
+    end
+  end
+
+  def wait_and_refresh(timeout, %User{} = a, %User{} = b) do
+    with :ok <- :timer.sleep(timeout),
+         %User{} = a <- Repo.get(User, a.id),
+         %User{} = b <- Repo.get(User, b.id) do
+      {:ok, a, b}
+    else
+      _e ->
+        :error
+    end
+  end
+
+  def parse_bio(bio, user \\ %User{info: %{source_data: %{}}})
+  def parse_bio(nil, _user), do: ""
+  def parse_bio(bio, _user) when bio == "", do: bio
+
+  def parse_bio(bio, user) do
+    mentions = Formatter.parse_mentions(bio)
+    tags = Formatter.parse_tags(bio)
+
+    emoji =
+      (user.info.source_data["tag"] || [])
+      |> Enum.filter(fn %{"type" => t} -> t == "Emoji" end)
+      |> Enum.map(fn %{"icon" => %{"url" => url}, "name" => name} ->
+        {String.trim(name, ":"), url}
+      end)
+
+    CommonUtils.format_input(bio, mentions, tags, "text/plain") |> Formatter.emojify(emoji)
+  end
+
+  def tag(user_identifiers, tags) when is_list(user_identifiers) do
+    Repo.transaction(fn ->
+      for user_identifier <- user_identifiers, do: tag(user_identifier, tags)
+    end)
+  end
+
+  def tag(nickname, tags) when is_binary(nickname),
+    do: tag(User.get_by_nickname(nickname), tags)
+
+  def tag(%User{} = user, tags),
+    do: update_tags(user, Enum.uniq(user.tags ++ normalize_tags(tags)))
+
+  def untag(user_identifiers, tags) when is_list(user_identifiers) do
+    Repo.transaction(fn ->
+      for user_identifier <- user_identifiers, do: untag(user_identifier, tags)
+    end)
+  end
+
+  def untag(nickname, tags) when is_binary(nickname),
+    do: untag(User.get_by_nickname(nickname), tags)
+
+  def untag(%User{} = user, tags), do: update_tags(user, user.tags -- normalize_tags(tags))
+
+  defp update_tags(%User{} = user, new_tags) do
+    {:ok, updated_user} =
+      user
+      |> change(%{tags: new_tags})
+      |> Repo.update()
+
+    updated_user
+  end
+
+  defp normalize_tags(tags) do
+    [tags]
+    |> List.flatten()
+    |> Enum.map(&String.downcase(&1))
+  end
 end

lib/pleroma/user/info.ex

@@ -0,0 +1,172 @@
+defmodule Pleroma.User.Info do
+  use Ecto.Schema
+  import Ecto.Changeset
+
+  embedded_schema do
+    field(:banner, :map, default: %{})
+    field(:background, :map, default: %{})
+    field(:source_data, :map, default: %{})
+    field(:note_count, :integer, default: 0)
+    field(:follower_count, :integer, default: 0)
+    field(:locked, :boolean, default: false)
+    field(:default_scope, :string, default: "public")
+    field(:blocks, {:array, :string}, default: [])
+    field(:domain_blocks, {:array, :string}, default: [])
+    field(:deactivated, :boolean, default: false)
+    field(:no_rich_text, :boolean, default: false)
+    field(:ap_enabled, :boolean, default: false)
+    field(:is_moderator, :boolean, default: false)
+    field(:is_admin, :boolean, default: false)
+    field(:keys, :string, default: nil)
+    field(:settings, :map, default: nil)
+    field(:magic_key, :string, default: nil)
+    field(:uri, :string, default: nil)
+    field(:topic, :string, default: nil)
+    field(:hub, :string, default: nil)
+    field(:salmon, :string, default: nil)
+    field(:hide_network, :boolean, default: false)
+
+    # Found in the wild
+    # ap_id -> Where is this used?
+    # bio -> Where is this used?
+    # avatar -> Where is this used?
+    # fqn -> Where is this used?
+    # host -> Where is this used?
+    # subject _> Where is this used?
+  end
+
+  def set_activation_status(info, deactivated) do
+    params = %{deactivated: deactivated}
+
+    info
+    |> cast(params, [:deactivated])
+    |> validate_required([:deactivated])
+  end
+
+  def add_to_note_count(info, number) do
+    set_note_count(info, info.note_count + number)
+  end
+
+  def set_note_count(info, number) do
+    params = %{note_count: Enum.max([0, number])}
+
+    info
+    |> cast(params, [:note_count])
+    |> validate_required([:note_count])
+  end
+
+  def set_follower_count(info, number) do
+    params = %{follower_count: Enum.max([0, number])}
+
+    info
+    |> cast(params, [:follower_count])
+    |> validate_required([:follower_count])
+  end
+
+  def set_blocks(info, blocks) do
+    params = %{blocks: blocks}
+
+    info
+    |> cast(params, [:blocks])
+    |> validate_required([:blocks])
+  end
+
+  def add_to_block(info, blocked) do
+    set_blocks(info, Enum.uniq([blocked | info.blocks]))
+  end
+
+  def remove_from_block(info, blocked) do
+    set_blocks(info, List.delete(info.blocks, blocked))
+  end
+
+  def set_domain_blocks(info, domain_blocks) do
+    params = %{domain_blocks: domain_blocks}
+
+    info
+    |> cast(params, [:domain_blocks])
+    |> validate_required([:domain_blocks])
+  end
+
+  def add_to_domain_block(info, domain_blocked) do
+    set_domain_blocks(info, Enum.uniq([domain_blocked | info.domain_blocks]))
+  end
+
+  def remove_from_domain_block(info, domain_blocked) do
+    set_domain_blocks(info, List.delete(info.domain_blocks, domain_blocked))
+  end
+
+  def set_keys(info, keys) do
+    params = %{keys: keys}
+
+    info
+    |> cast(params, [:keys])
+    |> validate_required([:keys])
+  end
+
+  def remote_user_creation(info, params) do
+    info
+    |> cast(params, [
+      :ap_enabled,
+      :source_data,
+      :banner,
+      :locked,
+      :magic_key,
+      :uri,
+      :hub,
+      :topic,
+      :salmon
+    ])
+  end
+
+  def user_upgrade(info, params) do
+    info
+    |> cast(params, [
+      :ap_enabled,
+      :source_data,
+      :banner,
+      :locked,
+      :magic_key
+    ])
+  end
+
+  def profile_update(info, params) do
+    info
+    |> cast(params, [
+      :locked,
+      :no_rich_text,
+      :default_scope,
+      :banner,
+      :hide_network,
+      :background
+    ])
+  end
+
+  def mastodon_profile_update(info, params) do
+    info
+    |> cast(params, [
+      :locked,
+      :banner
+    ])
+  end
+
+  def mastodon_settings_update(info, params) do
+    info
+    |> cast(params, [:settings])
+  end
+
+  def set_source_data(info, source_data) do
+    params = %{source_data: source_data}
+
+    info
+    |> cast(params, [:source_data])
+    |> validate_required([:source_data])
+  end
+
+  def admin_api_update(info, params) do
+    info
+    |> cast(params, [
+      :is_moderator,
+      :is_admin
+    ])
+  end
+end

lib/pleroma/user_invite_token.ex

@@ -3,7 +3,8 @@ defmodule Pleroma.UserInviteToken do
 
   import Ecto.Changeset
 
-  alias Pleroma.{User, UserInviteToken, Repo}
+  alias Pleroma.UserInviteToken
+  alias Pleroma.Repo
 
   schema "user_invite_tokens" do
     field(:token, :string)

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -10,8 +10,6 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   @httpoison Application.get_env(:pleroma, :httpoison)
 
-  @instance Application.get_env(:pleroma, :instance)
-
   # For Announce activities, we filter the recipients based on following status for any actors
   # that match actual users.  See issue #164 for more information about why this is necessary.
   defp get_recipients(%{"type" => "Announce"} = data) do
@@ -44,7 +42,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   defp check_actor_is_active(actor) do
     if not is_nil(actor) do
       with user <- User.get_cached_by_ap_id(actor),
-           nil <- user.info["deactivated"] do
+           false <- user.info.deactivated do
         :ok
       else
         _e -> :reject
@@ -273,8 +271,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
       "to" => [user.follower_address, "https://www.w3.org/ns/activitystreams#Public"]
     }
 
-    with Repo.delete(object),
-         Repo.delete_all(Activity.all_non_create_by_object_ap_id_q(id)),
+    with {:ok, _} <- Object.delete(object),
          {:ok, activity} <- insert(data, local),
          :ok <- maybe_federate(activity),
          {:ok, _actor} <- User.decrease_note_count(user) do
@@ -512,8 +509,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   end
 
   defp restrict_blocked(query, %{"blocking_user" => %User{info: info}}) do
-    blocks = info["blocks"] || []
-    domain_blocks = info["domain_blocks"] || []
+    blocks = info.blocks || []
+    domain_blocks = info.domain_blocks || []
 
     from(
       activity in query,
@@ -575,9 +572,17 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     |> Enum.reverse()
   end
 
-  def upload(file) do
-    data = Upload.store(file, Application.get_env(:pleroma, :instance)[:dedupe_media])
-    Repo.insert(%Object{data: data})
+  def upload(file, opts \\ []) do
+    with {:ok, data} <- Upload.store(file, opts) do
+      obj_data =
+        if opts[:actor] do
+          Map.put(data, "actor", opts[:actor])
+        else
+          data
+        end
+
+      Repo.insert(%Object{data: obj_data})
+    end
   end
 
   def user_data_from_user_object(data) do
@@ -628,9 +633,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   end
 
   def fetch_and_prepare_user_from_ap_id(ap_id) do
-    with {:ok, %{status_code: 200, body: body}} <-
-           @httpoison.get(ap_id, [Accept: "application/activity+json"], follow_redirect: true),
-         {:ok, data} <- Jason.decode(body) do
+    with {:ok, data} <- fetch_and_contain_remote_object_from_id(ap_id) do
       user_data_from_user_object(data)
     else
       e -> Logger.error("Could not decode user at fetch #{ap_id}, #{inspect(e)}")
@@ -657,14 +660,12 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     end
   end
 
-  @quarantined_instances Keyword.get(@instance, :quarantined_instances, [])
-
   def should_federate?(inbox, public) do
     if public do
       true
     else
       inbox_info = URI.parse(inbox)
-      inbox_info.host not in @quarantined_instances
+      !Enum.member?(Pleroma.Config.get([:instance, :quarantined_instances], []), inbox_info.host)
     end
   end
 
@@ -682,8 +683,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     remote_inboxes =
       (Pleroma.Web.Salmon.remote_users(activity) ++ followers)
       |> Enum.filter(fn user -> User.ap_enabled?(user) end)
-      |> Enum.map(fn %{info: %{"source_data" => data}} ->
-        (data["endpoints"] && data["endpoints"]["sharedInbox"]) || data["inbox"]
+      |> Enum.map(fn %{info: %{source_data: data}} ->
+        (is_map(data["endpoints"]) && Map.get(data["endpoints"], "sharedInbox")) || data["inbox"]
       end)
       |> Enum.uniq()
       |> Enum.filter(fn inbox -> should_federate?(inbox, public) end)
@@ -734,28 +735,22 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     else
       Logger.info("Fetching #{id} via AP")
 
-      with true <- String.starts_with?(id, "http"),
-           {:ok, %{body: body, status_code: code}} when code in 200..299 <-
-             @httpoison.get(
-               id,
-               [Accept: "application/activity+json"],
-               follow_redirect: true,
-               timeout: 10000,
-               recv_timeout: 20000
-             ),
-           {:ok, data} <- Jason.decode(body),
+      with {:ok, data} <- fetch_and_contain_remote_object_from_id(id),
            nil <- Object.normalize(data),
            params <- %{
              "type" => "Create",
              "to" => data["to"],
              "cc" => data["cc"],
-             "actor" => data["attributedTo"],
+             "actor" => data["actor"] || data["attributedTo"],
              "object" => data
            },
            :ok <- Transmogrifier.contain_origin(id, params),
            {:ok, activity} <- Transmogrifier.handle_incoming(params) do
         {:ok, Object.normalize(activity.data["object"])}
       else
+        {:error, {:reject, nil}} ->
+          {:reject, nil}
+
         object = %Object{} ->
           {:ok, object}
 
@@ -770,6 +765,24 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     end
   end
 
+  def fetch_and_contain_remote_object_from_id(id) do
+    Logger.info("Fetching #{id} via AP")
+
+    with true <- String.starts_with?(id, "http"),
+         {:ok, %{body: body, status: code}} when code in 200..299 <-
+           @httpoison.get(
+             id,
+             [{:Accept, "application/activity+json"}]
+           ),
+         {:ok, data} <- Jason.decode(body),
+         :ok <- Transmogrifier.contain_origin_from_id(id, data) do
+      {:ok, data}
+    else
+      e ->
+        {:error, e}
+    end
+  end
+
   def is_public?(activity) do
     "https://www.w3.org/ns/activitystreams#Public" in (activity.data["to"] ++
                                                          (activity.data["cc"] || []))
@@ -784,4 +797,38 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     y = activity.data["to"] ++ (activity.data["cc"] || [])
     visible_for_user?(activity, nil) || Enum.any?(x, &(&1 in y))
   end
+
+  # guard
+  def entire_thread_visible_for_user?(nil, _user), do: false
+
+  # child
+  def entire_thread_visible_for_user?(
+        %Activity{data: %{"object" => %{"inReplyTo" => parent_id}}} = tail,
+        user
+      )
+      when is_binary(parent_id) do
+    parent = Activity.get_in_reply_to_activity(tail)
+    visible_for_user?(tail, user) && entire_thread_visible_for_user?(parent, user)
+  end
+
+  # root
+  def entire_thread_visible_for_user?(tail, user), do: visible_for_user?(tail, user)
+
+  # filter out broken threads
+  def contain_broken_threads(%Activity{} = activity, %User{} = user) do
+    entire_thread_visible_for_user?(activity, user)
+  end
+
+  # do post-processing on a specific activity
+  def contain_activity(%Activity{} = activity, %User{} = user) do
+    contain_broken_threads(activity, user)
+  end
+
+  # do post-processing on a timeline
+  def contain_timeline(timeline, user) do
+    timeline
+    |> Enum.filter(fn activity ->
+      contain_activity(activity, user)
+    end)
+  end
 end

lib/pleroma/web/activity_pub/activity_pub_controller.ex

@@ -4,12 +4,27 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
   alias Pleroma.Web.ActivityPub.{ObjectView, UserView}
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Relay
+  alias Pleroma.Web.ActivityPub.Utils
   alias Pleroma.Web.Federator
 
   require Logger
 
   action_fallback(:errors)
 
+  plug(Pleroma.Web.FederatingPlug when action in [:inbox, :relay])
+  plug(:relay_active? when action in [:relay])
+
+  def relay_active?(conn, _) do
+    if Keyword.get(Application.get_env(:pleroma, :instance), :allow_relay) do
+      conn
+    else
+      conn
+      |> put_status(404)
+      |> json(%{error: "not found"})
+      |> halt
+    end
+  end
+
   def user(conn, %{"nickname" => nickname}) do
     with %User{} = user <- User.get_cached_by_nickname(nickname),
          {:ok, user} <- Pleroma.Web.WebFinger.ensure_keys_present(user) do
@@ -87,28 +102,46 @@ defmodule Pleroma.Web.ActivityPub.ActivityPubController do
     outbox(conn, %{"nickname" => nickname, "max_id" => nil})
   end
 
-  # TODO: Ensure that this inbox is a recipient of the message
+  def inbox(%{assigns: %{valid_signature: true}} = conn, %{"nickname" => nickname} = params) do
+    with %User{} = user <- User.get_cached_by_nickname(nickname),
+         true <- Utils.recipient_in_message(user.ap_id, params),
+         params <- Utils.maybe_splice_recipient(user.ap_id, params) do
+      Federator.enqueue(:incoming_ap_doc, params)
+      json(conn, "ok")
+    end
+  end
+
   def inbox(%{assigns: %{valid_signature: true}} = conn, params) do
     Federator.enqueue(:incoming_ap_doc, params)
     json(conn, "ok")
   end
 
+  # only accept relayed Creates
+  def inbox(conn, %{"type" => "Create"} = params) do
+    Logger.info(
+      "Signature missing or not from author, relayed Create message, fetching object from source"
+    )
+
+    ActivityPub.fetch_object_from_id(params["object"]["id"])
+
+    json(conn, "ok")
+  end
+
   def inbox(conn, params) do
     headers = Enum.into(conn.req_headers, %{})
 
-    if !String.contains?(headers["signature"] || "", params["actor"]) do
-      Logger.info("Signature not from author, relayed message, fetching from source")
-      ActivityPub.fetch_object_from_id(params["object"]["id"])
-    else
-      Logger.info("Signature error - make sure you are forwarding the HTTP Host header!")
-      Logger.info("Could not validate #{params["actor"]}")
+    if String.contains?(headers["signature"], params["actor"]) do
+      Logger.info(
+        "Signature validation error for: #{params["actor"]}, make sure you are forwarding the HTTP Host header!"
+      )
+
       Logger.info(inspect(conn.req_headers))
     end
 
-    json(conn, "ok")
+    json(conn, "error")
   end
 
-  def relay(conn, params) do
+  def relay(conn, _params) do
     with %User{} = user <- Relay.get_actor(),
          {:ok, user} <- Pleroma.Web.WebFinger.ensure_keys_present(user) do
       conn

lib/pleroma/web/activity_pub/mrf/ensure_re_prepended.ex

@@ -0,0 +1,40 @@
+defmodule Pleroma.Web.ActivityPub.MRF.EnsureRePrepended do
+  alias Pleroma.Object
+
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  @reply_prefix Regex.compile!("^re:[[:space:]]*", [:caseless])
+  def filter_by_summary(
+        %{"summary" => parent_summary} = _parent,
+        %{"summary" => child_summary} = child
+      )
+      when not is_nil(child_summary) and byte_size(child_summary) > 0 and
+             not is_nil(parent_summary) and byte_size(parent_summary) > 0 do
+    if (child_summary == parent_summary and not Regex.match?(@reply_prefix, child_summary)) or
+         (Regex.match?(@reply_prefix, parent_summary) &&
+            Regex.replace(@reply_prefix, parent_summary, "") == child_summary) do
+      Map.put(child, "summary", "re: " <> child_summary)
+    else
+      child
+    end
+  end
+
+  def filter_by_summary(_parent, child), do: child
+
+  def filter(%{"type" => activity_type} = object) when activity_type == "Create" do
+    child = object["object"]
+    in_reply_to = Object.normalize(child["inReplyTo"])
+
+    child =
+      if(in_reply_to,
+        do: filter_by_summary(in_reply_to.data, child),
+        else: child
+      )
+
+    object = Map.put(object, "object", child)
+
+    {:ok, object}
+  end
+
+  def filter(object), do: {:ok, object}
+end

lib/pleroma/web/activity_pub/mrf/normalize_markup.ex

@@ -0,0 +1,23 @@
+defmodule Pleroma.Web.ActivityPub.MRF.NormalizeMarkup do
+  alias Pleroma.HTML
+
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  def filter(%{"type" => activity_type} = object) when activity_type == "Create" do
+    scrub_policy = Pleroma.Config.get([:mrf_normalize_markup, :scrub_policy])
+
+    child = object["object"]
+
+    content =
+      child["content"]
+      |> HTML.filter_tags(scrub_policy)
+
+    child = Map.put(child, "content", content)
+
+    object = Map.put(object, "object", child)
+
+    {:ok, object}
+  end
+
+  def filter(object), do: {:ok, object}
+end

lib/pleroma/web/activity_pub/mrf/reject_non_public.ex

@@ -2,10 +2,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
   alias Pleroma.User
   @behaviour Pleroma.Web.ActivityPub.MRF
 
-  @mrf_rejectnonpublic Application.get_env(:pleroma, :mrf_rejectnonpublic)
-  @allow_followersonly Keyword.get(@mrf_rejectnonpublic, :allow_followersonly)
-  @allow_direct Keyword.get(@mrf_rejectnonpublic, :allow_direct)
-
   @impl true
   def filter(%{"type" => "Create"} = object) do
     user = User.get_cached_by_ap_id(object["actor"])
@@ -20,6 +16,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
         true -> "direct"
       end
 
+    policy = Pleroma.Config.get(:mrf_rejectnonpublic)
+
     case visibility do
       "public" ->
         {:ok, object}
@@ -28,14 +26,14 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
         {:ok, object}
 
       "followers" ->
-        with true <- @allow_followersonly do
+        with true <- Keyword.get(policy, :allow_followersonly) do
           {:ok, object}
         else
           _e -> {:reject, nil}
         end
 
       "direct" ->
-        with true <- @allow_direct do
+        with true <- Keyword.get(policy, :allow_direct) do
           {:ok, object}
         else
           _e -> {:reject, nil}

lib/pleroma/web/activity_pub/mrf/simple_policy.ex

@@ -2,60 +2,76 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
   alias Pleroma.User
   @behaviour Pleroma.Web.ActivityPub.MRF
 
-  @mrf_policy Application.get_env(:pleroma, :mrf_simple)
+  defp check_accept(%{host: actor_host} = _actor_info, object) do
+    accepts = Pleroma.Config.get([:mrf_simple, :accept])
 
-  @accept Keyword.get(@mrf_policy, :accept)
-  defp check_accept(%{host: actor_host} = actor_info, object)
-       when length(@accept) > 0 and not (actor_host in @accept) do
-    {:reject, nil}
+    cond do
+      accepts == [] -> {:ok, object}
+      actor_host == Pleroma.Config.get([Pleroma.Web.Endpoint, :url, :host]) -> {:ok, object}
+      Enum.member?(accepts, actor_host) -> {:ok, object}
+      true -> {:reject, nil}
+    end
   end
 
-  defp check_accept(actor_info, object), do: {:ok, object}
-
-  @reject Keyword.get(@mrf_policy, :reject)
-  defp check_reject(%{host: actor_host} = actor_info, object) when actor_host in @reject do
-    {:reject, nil}
+  defp check_reject(%{host: actor_host} = _actor_info, object) do
+    if Enum.member?(Pleroma.Config.get([:mrf_simple, :reject]), actor_host) do
+      {:reject, nil}
+    else
+      {:ok, object}
+    end
   end
 
-  defp check_reject(actor_info, object), do: {:ok, object}
+  defp check_media_removal(
+         %{host: actor_host} = _actor_info,
+         %{"type" => "Create", "object" => %{"attachment" => child_attachment}} = object
+       )
+       when length(child_attachment) > 0 do
+    object =
+      if Enum.member?(Pleroma.Config.get([:mrf_simple, :media_removal]), actor_host) do
+        child_object = Map.delete(object["object"], "attachment")
+        Map.put(object, "object", child_object)
+      else
+        object
+      end
 
-  @media_removal Keyword.get(@mrf_policy, :media_removal)
-  defp check_media_removal(%{host: actor_host} = actor_info, %{"type" => "Create"} = object)
-       when actor_host in @media_removal do
-    child_object = Map.delete(object["object"], "attachment")
-    object = Map.put(object, "object", child_object)
     {:ok, object}
   end
 
-  defp check_media_removal(actor_info, object), do: {:ok, object}
+  defp check_media_removal(_actor_info, object), do: {:ok, object}
 
-  @media_nsfw Keyword.get(@mrf_policy, :media_nsfw)
   defp check_media_nsfw(
-         %{host: actor_host} = actor_info,
+         %{host: actor_host} = _actor_info,
          %{
            "type" => "Create",
            "object" => %{"attachment" => child_attachment} = child_object
          } = object
        )
-       when actor_host in @media_nsfw and length(child_attachment) > 0 do
-    tags = (child_object["tag"] || []) ++ ["nsfw"]
-    child_object = Map.put(child_object, "tags", tags)
-    child_object = Map.put(child_object, "sensitive", true)
-    object = Map.put(object, "object", child_object)
+       when length(child_attachment) > 0 do
+    object =
+      if Enum.member?(Pleroma.Config.get([:mrf_simple, :media_nsfw]), actor_host) do
+        tags = (child_object["tag"] || []) ++ ["nsfw"]
+        child_object = Map.put(child_object, "tags", tags)
+        child_object = Map.put(child_object, "sensitive", true)
+        Map.put(object, "object", child_object)
+      else
+        object
+      end
+
     {:ok, object}
   end
 
-  defp check_media_nsfw(actor_info, object), do: {:ok, object}
+  defp check_media_nsfw(_actor_info, object), do: {:ok, object}
 
-  @ftl_removal Keyword.get(@mrf_policy, :federated_timeline_removal)
-  defp check_ftl_removal(%{host: actor_host} = actor_info, object)
-       when actor_host in @ftl_removal do
-    user = User.get_by_ap_id(object["actor"])
-
-    # flip to/cc relationship to make the post unlisted
+  defp check_ftl_removal(%{host: actor_host} = _actor_info, object) do
     object =
-      if "https://www.w3.org/ns/activitystreams#Public" in object["to"] and
-           user.follower_address in object["cc"] do
+      with true <-
+             Enum.member?(
+               Pleroma.Config.get([:mrf_simple, :federated_timeline_removal]),
+               actor_host
+             ),
+           user <- User.get_cached_by_ap_id(object["actor"]),
+           true <- "https://www.w3.org/ns/activitystreams#Public" in object["to"],
+           true <- user.follower_address in object["cc"] do
         to =
           List.delete(object["to"], "https://www.w3.org/ns/activitystreams#Public") ++
             [user.follower_address]
@@ -68,14 +84,12 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
         |> Map.put("to", to)
         |> Map.put("cc", cc)
       else
-        object
+        _ -> object
       end
 
     {:ok, object}
   end
 
-  defp check_ftl_removal(actor_info, object), do: {:ok, object}
-
   @impl true
   def filter(object) do
     actor_info = URI.parse(object["actor"])

lib/pleroma/web/activity_pub/mrf/user_allowlist.ex

@@ -0,0 +1,23 @@
+defmodule Pleroma.Web.ActivityPub.MRF.UserAllowListPolicy do
+  alias Pleroma.Config
+
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  defp filter_by_list(object, []), do: {:ok, object}
+
+  defp filter_by_list(%{"actor" => actor} = object, allow_list) do
+    if actor in allow_list do
+      {:ok, object}
+    else
+      {:reject, nil}
+    end
+  end
+
+  @impl true
+  def filter(object) do
+    actor_info = URI.parse(object["actor"])
+    allow_list = Config.get([:mrf_user_allowlist, String.to_atom(actor_info.host)], [])
+
+    filter_by_list(object, allow_list)
+  end
+end

lib/pleroma/web/activity_pub/relay.ex

@@ -12,11 +12,12 @@ defmodule Pleroma.Web.ActivityPub.Relay do
          %User{} = target_user <- User.get_or_fetch_by_ap_id(target_instance),
          {:ok, activity} <- ActivityPub.follow(local_user, target_user) do
       Logger.info("relay: followed instance: #{target_instance}; id=#{activity.data["id"]}")
+      {:ok, activity}
     else
-      e -> Logger.error("error: #{inspect(e)}")
+      e ->
+        Logger.error("error: #{inspect(e)}")
+        {:error, e}
     end
-
-    :ok
   end
 
   def unfollow(target_instance) do
@@ -24,11 +25,12 @@ defmodule Pleroma.Web.ActivityPub.Relay do
          %User{} = target_user <- User.get_or_fetch_by_ap_id(target_instance),
          {:ok, activity} <- ActivityPub.unfollow(local_user, target_user) do
       Logger.info("relay: unfollowed instance: #{target_instance}: id=#{activity.data["id"]}")
+      {:ok, activity}
     else
-      e -> Logger.error("error: #{inspect(e)}")
+      e ->
+        Logger.error("error: #{inspect(e)}")
+        {:error, e}
     end
-
-    :ok
   end
 
   def publish(%Activity{data: %{"type" => "Create"}} = activity) do

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -21,19 +21,25 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     if is_binary(Enum.at(actor, 0)) do
       Enum.at(actor, 0)
     else
-      Enum.find(actor, fn %{"type" => type} -> type == "Person" end)
+      Enum.find(actor, fn %{"type" => type} -> type in ["Person", "Service", "Application"] end)
       |> Map.get("id")
     end
   end
 
-  def get_actor(%{"actor" => actor}) when is_map(actor) do
-    actor["id"]
+  def get_actor(%{"actor" => %{"id" => id}}) when is_bitstring(id) do
+    id
+  end
+
+  def get_actor(%{"actor" => nil, "attributedTo" => actor}) when not is_nil(actor) do
+    get_actor(%{"actor" => actor})
   end
 
   @doc """
   Checks that an imported AP object's actor matches the domain it came from.
   """
-  def contain_origin(id, %{"actor" => actor} = params) do
+  def contain_origin(_id, %{"actor" => nil}), do: :error
+
+  def contain_origin(id, %{"actor" => _actor} = params) do
     id_uri = URI.parse(id)
     actor_uri = URI.parse(get_actor(params))
 
@@ -44,6 +50,19 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     end
   end
 
+  def contain_origin_from_id(_id, %{"id" => nil}), do: :error
+
+  def contain_origin_from_id(id, %{"id" => other_id} = _params) do
+    id_uri = URI.parse(id)
+    other_uri = URI.parse(other_id)
+
+    if id_uri.host == other_uri.host do
+      :ok
+    else
+      :error
+    end
+  end
+
   @doc """
   Modifies an incoming AP object (mastodon format) to our internal format.
   """
@@ -51,6 +70,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     object
     |> fix_actor
     |> fix_attachments
+    |> fix_url
     |> fix_context
     |> fix_in_reply_to
     |> fix_emoji
@@ -96,9 +116,25 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     object
   end
 
-  def fix_in_reply_to(%{"inReplyTo" => in_reply_to_id} = object)
-      when not is_nil(in_reply_to_id) do
-    case ActivityPub.fetch_object_from_id(in_reply_to_id) do
+  def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object)
+      when not is_nil(in_reply_to) do
+    in_reply_to_id =
+      cond do
+        is_bitstring(in_reply_to) ->
+          in_reply_to
+
+        is_map(in_reply_to) && is_bitstring(in_reply_to["id"]) ->
+          in_reply_to["id"]
+
+        is_list(in_reply_to) && is_bitstring(Enum.at(in_reply_to, 0)) ->
+          Enum.at(in_reply_to, 0)
+
+        # Maybe I should output an error too?
+        true ->
+          ""
+      end
+
+    case fetch_obj_helper(in_reply_to_id) do
       {:ok, replied_object} ->
         with %Activity{} = activity <-
                Activity.get_create_activity_by_object_ap_id(replied_object.data["id"]) do
@@ -110,12 +146,12 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
           |> Map.put("context", replied_object.data["context"] || object["conversation"])
         else
           e ->
-            Logger.error("Couldn't fetch #{object["inReplyTo"]} #{inspect(e)}")
+            Logger.error("Couldn't fetch \"#{inspect(in_reply_to_id)}\", error: #{inspect(e)}")
             object
         end
 
       e ->
-        Logger.error("Couldn't fetch #{object["inReplyTo"]} #{inspect(e)}")
+        Logger.error("Couldn't fetch \"#{inspect(in_reply_to_id)}\", error: #{inspect(e)}")
         object
     end
   end
@@ -130,9 +166,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> Map.put("conversation", context)
   end
 
-  def fix_attachments(object) do
+  def fix_attachments(%{"attachment" => attachment} = object) when is_list(attachment) do
     attachments =
-      (object["attachment"] || [])
+      attachment
       |> Enum.map(fn data ->
         url = [%{"type" => "Link", "mediaType" => data["mediaType"], "href" => data["url"]}]
         Map.put(data, "url", url)
@@ -142,21 +178,41 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> Map.put("attachment", attachments)
   end
 
-  def fix_emoji(object) do
-    tags = object["tag"] || []
+  def fix_attachments(%{"attachment" => attachment} = object) when is_map(attachment) do
+    Map.put(object, "attachment", [attachment])
+    |> fix_attachments()
+  end
+
+  def fix_attachments(object), do: object
+
+  def fix_url(%{"url" => url} = object) when is_map(url) do
+    object
+    |> Map.put("url", url["href"])
+  end
+
+  def fix_url(%{"url" => url} = object) when is_list(url) do
+    first_element = Enum.at(url, 0)
+
+    url_string =
+      cond do
+        is_bitstring(first_element) -> first_element
+        is_map(first_element) -> first_element["href"] || ""
+        true -> ""
+      end
+
+    object
+    |> Map.put("url", url_string)
+  end
+
+  def fix_url(object), do: object
+
+  def fix_emoji(%{"tag" => tags} = object) when is_list(tags) do
     emoji = tags |> Enum.filter(fn data -> data["type"] == "Emoji" and data["icon"] end)
 
     emoji =
       emoji
       |> Enum.reduce(%{}, fn data, mapping ->
-        name = data["name"]
-
-        name =
-          if String.starts_with?(name, ":") do
-            name |> String.slice(1..-2)
-          else
-            name
-          end
+        name = String.trim(data["name"], ":")
 
         mapping |> Map.put(name, data["icon"]["url"])
       end)
@@ -168,18 +224,37 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> Map.put("emoji", emoji)
   end
 
-  def fix_tag(object) do
+  def fix_emoji(%{"tag" => %{"type" => "Emoji"} = tag} = object) do
+    name = String.trim(tag["name"], ":")
+    emoji = %{name => tag["icon"]["url"]}
+
+    object
+    |> Map.put("emoji", emoji)
+  end
+
+  def fix_emoji(object), do: object
+
+  def fix_tag(%{"tag" => tag} = object) when is_list(tag) do
     tags =
-      (object["tag"] || [])
+      tag
       |> Enum.filter(fn data -> data["type"] == "Hashtag" and data["name"] end)
       |> Enum.map(fn data -> String.slice(data["name"], 1..-1) end)
 
-    combined = (object["tag"] || []) ++ tags
+    combined = tag ++ tags
 
     object
     |> Map.put("tag", combined)
   end
 
+  def fix_tag(%{"tag" => %{"type" => "Hashtag", "name" => hashtag} = tag} = object) do
+    combined = [tag, String.slice(hashtag, 1..-1)]
+
+    object
+    |> Map.put("tag", combined)
+  end
+
+  def fix_tag(object), do: object
+
   # content map usually only has one language so this will do for now.
   def fix_content_map(%{"contentMap" => content_map} = object) do
     content_groups = Map.to_list(content_map)
@@ -191,6 +266,32 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def fix_content_map(object), do: object
 
+  defp mastodon_follow_hack(%{"id" => id, "actor" => follower_id}, followed) do
+    with true <- id =~ "follows",
+         %User{local: true} = follower <- User.get_cached_by_ap_id(follower_id),
+         %Activity{} = activity <- Utils.fetch_latest_follow(follower, followed) do
+      {:ok, activity}
+    else
+      _ -> {:error, nil}
+    end
+  end
+
+  defp mastodon_follow_hack(_, _), do: {:error, nil}
+
+  defp get_follow_activity(follow_object, followed) do
+    with object_id when not is_nil(object_id) <- Utils.get_ap_id(follow_object),
+         {_, %Activity{} = activity} <- {:activity, Activity.get_by_ap_id(object_id)} do
+      {:ok, activity}
+    else
+      # Can't find the activity. This might a Mastodon 2.3 "Accept"
+      {:activity, nil} ->
+        mastodon_follow_hack(follow_object, followed)
+
+      _ ->
+        {:error, nil}
+    end
+  end
+
   # disallow objects with bogus IDs
   def handle_incoming(%{"id" => nil}), do: :error
   def handle_incoming(%{"id" => ""}), do: :error
@@ -201,7 +302,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   # - tags
   # - emoji
   def handle_incoming(%{"type" => "Create", "object" => %{"type" => objtype} = object} = data)
-      when objtype in ["Article", "Note", "Video"] do
+      when objtype in ["Article", "Note", "Video", "Page"] do
     actor = get_actor(data)
 
     data =
@@ -256,37 +357,13 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     end
   end
 
-  defp mastodon_follow_hack(%{"id" => id, "actor" => follower_id}, followed) do
-    with true <- id =~ "follows",
-         %User{local: true} = follower <- User.get_cached_by_ap_id(follower_id),
-         %Activity{} = activity <- Utils.fetch_latest_follow(follower, followed) do
-      {:ok, activity}
-    else
-      _ -> {:error, nil}
-    end
-  end
-
-  defp mastodon_follow_hack(_), do: {:error, nil}
-
-  defp get_follow_activity(follow_object, followed) do
-    with object_id when not is_nil(object_id) <- Utils.get_ap_id(follow_object),
-         {_, %Activity{} = activity} <- {:activity, Activity.get_by_ap_id(object_id)} do
-      {:ok, activity}
-    else
-      # Can't find the activity. This might a Mastodon 2.3 "Accept"
-      {:activity, nil} ->
-        mastodon_follow_hack(follow_object, followed)
-
-      _ ->
-        {:error, nil}
-    end
-  end
-
   def handle_incoming(
-        %{"type" => "Accept", "object" => follow_object, "actor" => actor, "id" => id} = data
+        %{"type" => "Accept", "object" => follow_object, "actor" => _actor, "id" => _id} = data
       ) do
-    with %User{} = followed <- User.get_or_fetch_by_ap_id(actor),
+    with actor <- get_actor(data),
+         %User{} = followed <- User.get_or_fetch_by_ap_id(actor),
          {:ok, follow_activity} <- get_follow_activity(follow_object, followed),
+         {:ok, follow_activity} <- Utils.update_follow_state(follow_activity, "accept"),
          %User{local: true} = follower <- User.get_cached_by_ap_id(follow_activity.data["actor"]),
          {:ok, activity} <-
            ActivityPub.accept(%{
@@ -297,7 +374,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
              local: false
            }) do
       if not User.following?(follower, followed) do
-        {:ok, follower} = User.follow(follower, followed)
+        {:ok, _follower} = User.follow(follower, followed)
       end
 
       {:ok, activity}
@@ -307,10 +384,12 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   end
 
   def handle_incoming(
-        %{"type" => "Reject", "object" => follow_object, "actor" => actor, "id" => id} = data
+        %{"type" => "Reject", "object" => follow_object, "actor" => _actor, "id" => _id} = data
       ) do
-    with %User{} = followed <- User.get_or_fetch_by_ap_id(actor),
+    with actor <- get_actor(data),
+         %User{} = followed <- User.get_or_fetch_by_ap_id(actor),
          {:ok, follow_activity} <- get_follow_activity(follow_object, followed),
+         {:ok, follow_activity} <- Utils.update_follow_state(follow_activity, "reject"),
          %User{local: true} = follower <- User.get_cached_by_ap_id(follow_activity.data["actor"]),
          {:ok, activity} <-
            ActivityPub.accept(%{
@@ -329,11 +408,11 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   end
 
   def handle_incoming(
-        %{"type" => "Like", "object" => object_id, "actor" => actor, "id" => id} = _data
+        %{"type" => "Like", "object" => object_id, "actor" => _actor, "id" => id} = data
       ) do
-    with %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
-         {:ok, object} <-
-           get_obj_helper(object_id) || ActivityPub.fetch_object_from_id(object_id),
+    with actor <- get_actor(data),
+         %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
+         {:ok, object} <- get_obj_helper(object_id) || fetch_obj_helper(object_id),
          {:ok, activity, _object} <- ActivityPub.like(actor, object, id, false) do
       {:ok, activity}
     else
@@ -342,11 +421,11 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   end
 
   def handle_incoming(
-        %{"type" => "Announce", "object" => object_id, "actor" => actor, "id" => id} = _data
+        %{"type" => "Announce", "object" => object_id, "actor" => _actor, "id" => id} = data
       ) do
-    with %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
-         {:ok, object} <-
-           get_obj_helper(object_id) || ActivityPub.fetch_object_from_id(object_id),
+    with actor <- get_actor(data),
+         %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
+         {:ok, object} <- get_obj_helper(object_id) || fetch_obj_helper(object_id),
          {:ok, activity, _object} <- ActivityPub.announce(actor, object, id, false) do
       {:ok, activity}
     else
@@ -368,7 +447,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
       update_data =
         new_user_data
         |> Map.take([:name, :bio, :avatar])
-        |> Map.put(:info, Map.merge(actor.info, %{"banner" => banner, "locked" => locked}))
+        |> Map.put(:info, %{"banner" => banner, "locked" => locked})
 
       actor
       |> User.upgrade_changeset(update_data)
@@ -388,15 +467,20 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     end
   end
 
-  # TODO: Make secure.
+  # TODO: We presently assume that any actor on the same origin domain as the object being
+  # deleted has the rights to delete that object.  A better way to validate whether or not
+  # the object should be deleted is to refetch the object URI, which should return either
+  # an error or a tombstone.  This would allow us to verify that a deletion actually took
+  # place.
   def handle_incoming(
-        %{"type" => "Delete", "object" => object_id, "actor" => actor, "id" => _id} = _data
+        %{"type" => "Delete", "object" => object_id, "actor" => _actor, "id" => _id} = data
       ) do
     object_id = Utils.get_ap_id(object_id)
 
-    with %User{} = _actor <- User.get_or_fetch_by_ap_id(actor),
-         {:ok, object} <-
-           get_obj_helper(object_id) || ActivityPub.fetch_object_from_id(object_id),
+    with actor <- get_actor(data),
+         %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
+         {:ok, object} <- get_obj_helper(object_id) || fetch_obj_helper(object_id),
+         :ok <- contain_origin(actor.ap_id, object.data),
          {:ok, activity} <- ActivityPub.delete(object, false) do
       {:ok, activity}
     else
@@ -408,13 +492,13 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         %{
           "type" => "Undo",
           "object" => %{"type" => "Announce", "object" => object_id},
-          "actor" => actor,
+          "actor" => _actor,
           "id" => id
-        } = _data
+        } = data
       ) do
-    with %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
-         {:ok, object} <-
-           get_obj_helper(object_id) || ActivityPub.fetch_object_from_id(object_id),
+    with actor <- get_actor(data),
+         %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
+         {:ok, object} <- get_obj_helper(object_id) || fetch_obj_helper(object_id),
          {:ok, activity, _} <- ActivityPub.unannounce(actor, object, id, false) do
       {:ok, activity}
     else
@@ -436,13 +520,10 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
       User.unfollow(follower, followed)
       {:ok, activity}
     else
-      e -> :error
+      _e -> :error
     end
   end
 
-  @ap_config Application.get_env(:pleroma, :activitypub)
-  @accept_blocks Keyword.get(@ap_config, :accept_blocks)
-
   def handle_incoming(
         %{
           "type" => "Undo",
@@ -451,21 +532,21 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
           "id" => id
         } = _data
       ) do
-    with true <- @accept_blocks,
+    with true <- Pleroma.Config.get([:activitypub, :accept_blocks]),
          %User{local: true} = blocked <- User.get_cached_by_ap_id(blocked),
          %User{} = blocker <- User.get_or_fetch_by_ap_id(blocker),
          {:ok, activity} <- ActivityPub.unblock(blocker, blocked, id, false) do
       User.unblock(blocker, blocked)
       {:ok, activity}
     else
-      e -> :error
+      _e -> :error
     end
   end
 
   def handle_incoming(
-        %{"type" => "Block", "object" => blocked, "actor" => blocker, "id" => id} = data
+        %{"type" => "Block", "object" => blocked, "actor" => blocker, "id" => id} = _data
       ) do
-    with true <- @accept_blocks,
+    with true <- Pleroma.Config.get([:activitypub, :accept_blocks]),
          %User{local: true} = blocked = User.get_cached_by_ap_id(blocked),
          %User{} = blocker = User.get_or_fetch_by_ap_id(blocker),
          {:ok, activity} <- ActivityPub.block(blocker, blocked, id, false) do
@@ -473,7 +554,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
       User.block(blocker, blocked)
       {:ok, activity}
     else
-      e -> :error
+      _e -> :error
     end
   end
 
@@ -481,13 +562,13 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         %{
           "type" => "Undo",
           "object" => %{"type" => "Like", "object" => object_id},
-          "actor" => actor,
+          "actor" => _actor,
           "id" => id
-        } = _data
+        } = data
       ) do
-    with %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
-         {:ok, object} <-
-           get_obj_helper(object_id) || ActivityPub.fetch_object_from_id(object_id),
+    with actor <- get_actor(data),
+         %User{} = actor <- User.get_or_fetch_by_ap_id(actor),
+         {:ok, object} <- get_obj_helper(object_id) || fetch_obj_helper(object_id),
          {:ok, activity, _, _} <- ActivityPub.unlike(actor, object, id, false) do
       {:ok, activity}
     else
@@ -497,6 +578,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def handle_incoming(_), do: :error
 
+  def fetch_obj_helper(id) when is_bitstring(id), do: ActivityPub.fetch_object_from_id(id)
+  def fetch_obj_helper(obj) when is_map(obj), do: ActivityPub.fetch_object_from_id(obj["id"])
+
   def get_obj_helper(id) do
     if object = Object.normalize(id), do: {:ok, object}, else: nil
   end
@@ -523,6 +607,8 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> prepare_attachments
     |> set_conversation
     |> set_reply_to_uri
+    |> strip_internal_fields
+    |> strip_internal_tags
   end
 
   #  @doc
@@ -538,7 +624,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     data =
       data
       |> Map.put("object", object)
-      |> Map.put("@context", "https://www.w3.org/ns/activitystreams")
+      |> Map.merge(Utils.make_json_ld_header())
 
     {:ok, data}
   end
@@ -557,7 +643,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
       data =
         data
         |> Map.put("object", object)
-        |> Map.put("@context", "https://www.w3.org/ns/activitystreams")
+        |> Map.merge(Utils.make_json_ld_header())
 
       {:ok, data}
     end
@@ -575,7 +661,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
       data =
         data
         |> Map.put("object", object)
-        |> Map.put("@context", "https://www.w3.org/ns/activitystreams")
+        |> Map.merge(Utils.make_json_ld_header())
 
       {:ok, data}
     end
@@ -585,14 +671,14 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     data =
       data
       |> maybe_fix_object_url
-      |> Map.put("@context", "https://www.w3.org/ns/activitystreams")
+      |> Map.merge(Utils.make_json_ld_header())
 
     {:ok, data}
   end
 
   def maybe_fix_object_url(data) do
     if is_binary(data["object"]) and not String.starts_with?(data["object"], "http") do
-      case ActivityPub.fetch_object_from_id(data["object"]) do
+      case fetch_obj_helper(data["object"]) do
         {:ok, relative_object} ->
           if relative_object.data["external_url"] do
             _data =
@@ -627,12 +713,9 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   end
 
   def add_mention_tags(object) do
-    recipients = object["to"] ++ (object["cc"] || [])
-
     mentions =
-      recipients
-      |> Enum.map(fn ap_id -> User.get_cached_by_ap_id(ap_id) end)
-      |> Enum.filter(& &1)
+      object
+      |> Utils.get_notified_from_object()
       |> Enum.map(fn user ->
         %{"type" => "Mention", "href" => user.ap_id, "name" => "@#{user.nickname}"}
       end)
@@ -692,6 +775,29 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> Map.put("attachment", attachments)
   end
 
+  defp strip_internal_fields(object) do
+    object
+    |> Map.drop([
+      "likes",
+      "like_count",
+      "announcements",
+      "announcement_count",
+      "emoji",
+      "context_id"
+    ])
+  end
+
+  defp strip_internal_tags(%{"tag" => tags} = object) do
+    tags =
+      tags
+      |> Enum.filter(fn x -> is_map(x) end)
+
+    object
+    |> Map.put("tag", tags)
+  end
+
+  defp strip_internal_tags(object), do: object
+
   defp user_upgrade_task(user) do
     old_follower_address = User.ap_followers(user)
 
@@ -744,10 +850,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   def upgrade_user_from_ap_id(ap_id, async \\ true) do
     with %User{local: false} = user <- User.get_by_ap_id(ap_id),
          {:ok, data} <- ActivityPub.fetch_and_prepare_user_from_ap_id(ap_id) do
-      data =
-        data
-        |> Map.put(:info, Map.merge(user.info, data[:info]))
-
       already_ap = User.ap_enabled?(user)
 
       {:ok, user} =

lib/pleroma/web/activity_pub/utils.ex

@@ -1,11 +1,13 @@
 defmodule Pleroma.Web.ActivityPub.Utils do
-  alias Pleroma.{Repo, Web, Object, Activity, User}
+  alias Pleroma.{Repo, Web, Object, Activity, User, Notification}
   alias Pleroma.Web.Router.Helpers
   alias Pleroma.Web.Endpoint
   alias Ecto.{Changeset, UUID}
   import Ecto.Query
   require Logger
 
+  @supported_object_types ["Article", "Note", "Video", "Page"]
+
   # Some implementations send the actor URI as the actor field, others send the entire actor object,
   # so figure out what the actor's URI is based on what we have.
   def get_ap_id(object) do
@@ -19,22 +21,58 @@ defmodule Pleroma.Web.ActivityPub.Utils do
     Map.put(params, "actor", get_ap_id(params["actor"]))
   end
 
+  defp recipient_in_collection(ap_id, coll) when is_binary(coll), do: ap_id == coll
+  defp recipient_in_collection(ap_id, coll) when is_list(coll), do: ap_id in coll
+  defp recipient_in_collection(_, _), do: false
+
+  def recipient_in_message(ap_id, params) do
+    cond do
+      recipient_in_collection(ap_id, params["to"]) ->
+        true
+
+      recipient_in_collection(ap_id, params["cc"]) ->
+        true
+
+      recipient_in_collection(ap_id, params["bto"]) ->
+        true
+
+      recipient_in_collection(ap_id, params["bcc"]) ->
+        true
+
+      # if the message is unaddressed at all, then assume it is directly addressed
+      # to the recipient
+      !params["to"] && !params["cc"] && !params["bto"] && !params["bcc"] ->
+        true
+
+      true ->
+        false
+    end
+  end
+
+  defp extract_list(target) when is_binary(target), do: [target]
+  defp extract_list(lst) when is_list(lst), do: lst
+  defp extract_list(_), do: []
+
+  def maybe_splice_recipient(ap_id, params) do
+    need_splice =
+      !recipient_in_collection(ap_id, params["to"]) &&
+        !recipient_in_collection(ap_id, params["cc"])
+
+    cc_list = extract_list(params["cc"])
+
+    if need_splice do
+      params
+      |> Map.put("cc", [ap_id | cc_list])
+    else
+      params
+    end
+  end
+
   def make_json_ld_header do
     %{
       "@context" => [
         "https://www.w3.org/ns/activitystreams",
-        "https://w3id.org/security/v1",
-        %{
-          "manuallyApprovesFollowers" => "as:manuallyApprovesFollowers",
-          "sensitive" => "as:sensitive",
-          "Hashtag" => "as:Hashtag",
-          "ostatus" => "http://ostatus.org#",
-          "atomUri" => "ostatus:atomUri",
-          "inReplyToAtomUri" => "ostatus:inReplyToAtomUri",
-          "conversation" => "ostatus:conversation",
-          "toot" => "http://joinmastodon.org/ns#",
-          "Emoji" => "toot:Emoji"
-        }
+        "#{Web.base_url()}/schemas/litepub-0.1.jsonld"
       ]
     }
   end
@@ -59,6 +97,21 @@ defmodule Pleroma.Web.ActivityPub.Utils do
     "#{Web.base_url()}/#{type}/#{UUID.generate()}"
   end
 
+  def get_notified_from_object(%{"type" => type} = object) when type in @supported_object_types do
+    fake_create_activity = %{
+      "to" => object["to"],
+      "cc" => object["cc"],
+      "type" => "Create",
+      "object" => object
+    }
+
+    Notification.get_notified_from_activity(%Activity{data: fake_create_activity}, false)
+  end
+
+  def get_notified_from_object(object) do
+    Notification.get_notified_from_activity(%Activity{data: object}, false)
+  end
+
   def create_context(context) do
     context = context || generate_id("contexts")
     changeset = Object.context_mapping(context)
@@ -128,7 +181,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
   Inserts a full object if it is contained in an activity.
   """
   def insert_full_object(%{"object" => %{"type" => type} = object_data})
-      when is_map(object_data) and type in ["Article", "Note", "Video"] do
+      when is_map(object_data) and type in @supported_object_types do
     with {:ok, _} <- Object.create(object_data) do
       :ok
     end
@@ -239,7 +292,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
   """
   def make_follow_data(
         %User{ap_id: follower_id},
-        %User{ap_id: followed_id} = followed,
+        %User{ap_id: followed_id} = _followed,
         activity_id
       ) do
     data = %{
@@ -247,11 +300,11 @@ defmodule Pleroma.Web.ActivityPub.Utils do
       "actor" => follower_id,
       "to" => [followed_id],
       "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
-      "object" => followed_id
+      "object" => followed_id,
+      "state" => "pending"
     }
 
     data = if activity_id, do: Map.put(data, "id", activity_id), else: data
-    data = if User.locked?(followed), do: Map.put(data, "state", "pending"), else: data
 
     data
   end

lib/pleroma/web/activity_pub/views/object_view.ex

@@ -1,27 +1,34 @@
 defmodule Pleroma.Web.ActivityPub.ObjectView do
   use Pleroma.Web, :view
+  alias Pleroma.{Object, Activity}
   alias Pleroma.Web.ActivityPub.Transmogrifier
 
-  def render("object.json", %{object: object}) do
-    base = %{
-      "@context" => [
-        "https://www.w3.org/ns/activitystreams",
-        "https://w3id.org/security/v1",
-        %{
-          "manuallyApprovesFollowers" => "as:manuallyApprovesFollowers",
-          "sensitive" => "as:sensitive",
-          "Hashtag" => "as:Hashtag",
-          "ostatus" => "http://ostatus.org#",
-          "atomUri" => "ostatus:atomUri",
-          "inReplyToAtomUri" => "ostatus:inReplyToAtomUri",
-          "conversation" => "ostatus:conversation",
-          "toot" => "http://joinmastodon.org/ns#",
-          "Emoji" => "toot:Emoji"
-        }
-      ]
-    }
+  def render("object.json", %{object: %Object{} = object}) do
+    base = Pleroma.Web.ActivityPub.Utils.make_json_ld_header()
 
     additional = Transmogrifier.prepare_object(object.data)
     Map.merge(base, additional)
   end
+
+  def render("object.json", %{object: %Activity{data: %{"type" => "Create"}} = activity}) do
+    base = Pleroma.Web.ActivityPub.Utils.make_json_ld_header()
+    object = Object.normalize(activity.data["object"])
+
+    additional =
+      Transmogrifier.prepare_object(activity.data)
+      |> Map.put("object", Transmogrifier.prepare_object(object.data))
+
+    Map.merge(base, additional)
+  end
+
+  def render("object.json", %{object: %Activity{} = activity}) do
+    base = Pleroma.Web.ActivityPub.Utils.make_json_ld_header()
+    object = Object.normalize(activity.data["object"])
+
+    additional =
+      Transmogrifier.prepare_object(activity.data)
+      |> Map.put("object", object.data["id"])
+
+    Map.merge(base, additional)
+  end
 end

lib/pleroma/web/activity_pub/views/user_view.ex

@@ -12,12 +12,11 @@ defmodule Pleroma.Web.ActivityPub.UserView do
   # the instance itself is not a Person, but instead an Application
   def render("user.json", %{user: %{nickname: nil} = user}) do
     {:ok, user} = WebFinger.ensure_keys_present(user)
-    {:ok, _, public_key} = Salmon.keys_from_pem(user.info["keys"])
+    {:ok, _, public_key} = Salmon.keys_from_pem(user.info.keys)
     public_key = :public_key.pem_entry_encode(:SubjectPublicKeyInfo, public_key)
     public_key = :public_key.pem_encode([public_key])
 
     %{
-      "@context" => "https://www.w3.org/ns/activitystreams",
       "id" => user.ap_id,
       "type" => "Application",
       "following" => "#{user.ap_id}/following",
@@ -36,11 +35,12 @@ defmodule Pleroma.Web.ActivityPub.UserView do
         "sharedInbox" => "#{Pleroma.Web.Endpoint.url()}/inbox"
       }
     }
+    |> Map.merge(Utils.make_json_ld_header())
   end
 
   def render("user.json", %{user: user}) do
     {:ok, user} = WebFinger.ensure_keys_present(user)
-    {:ok, _, public_key} = Salmon.keys_from_pem(user.info["keys"])
+    {:ok, _, public_key} = Salmon.keys_from_pem(user.info.keys)
     public_key = :public_key.pem_entry_encode(:SubjectPublicKeyInfo, public_key)
     public_key = :public_key.pem_encode([public_key])
 
@@ -55,7 +55,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       "name" => user.name,
       "summary" => user.bio,
       "url" => user.ap_id,
-      "manuallyApprovesFollowers" => user.info["locked"] || false,
+      "manuallyApprovesFollowers" => user.info.locked,
       "publicKey" => %{
         "id" => "#{user.ap_id}#main-key",
         "owner" => user.ap_id,
@@ -72,7 +72,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
         "type" => "Image",
         "url" => User.banner_url(user)
       },
-      "tag" => user.info["source_data"]["tag"] || []
+      "tag" => user.info.source_data["tag"] || []
     }
     |> Map.merge(Utils.make_json_ld_header())
   end
@@ -82,7 +82,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
     query = from(user in query, select: [:ap_id])
     following = Repo.all(query)
 
-    collection(following, "#{user.ap_id}/following", page)
+    collection(following, "#{user.ap_id}/following", page, !user.info.hide_network)
     |> Map.merge(Utils.make_json_ld_header())
   end
 
@@ -95,7 +95,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       "id" => "#{user.ap_id}/following",
       "type" => "OrderedCollection",
       "totalItems" => length(following),
-      "first" => collection(following, "#{user.ap_id}/following", 1)
+      "first" => collection(following, "#{user.ap_id}/following", 1, !user.info.hide_network)
     }
     |> Map.merge(Utils.make_json_ld_header())
   end
@@ -105,7 +105,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
     query = from(user in query, select: [:ap_id])
     followers = Repo.all(query)
 
-    collection(followers, "#{user.ap_id}/followers", page)
+    collection(followers, "#{user.ap_id}/followers", page, !user.info.hide_network)
     |> Map.merge(Utils.make_json_ld_header())
   end
 
@@ -118,7 +118,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       "id" => "#{user.ap_id}/followers",
       "type" => "OrderedCollection",
       "totalItems" => length(followers),
-      "first" => collection(followers, "#{user.ap_id}/followers", 1)
+      "first" => collection(followers, "#{user.ap_id}/followers", 1, !user.info.hide_network)
     }
     |> Map.merge(Utils.make_json_ld_header())
   end
@@ -172,7 +172,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
     end
   end
 
-  def collection(collection, iri, page, total \\ nil) do
+  def collection(collection, iri, page, show_items \\ true, total \\ nil) do
     offset = (page - 1) * 10
     items = Enum.slice(collection, offset, 10)
     items = Enum.map(items, fn user -> user.ap_id end)
@@ -183,7 +183,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       "type" => "OrderedCollectionPage",
       "partOf" => iri,
       "totalItems" => total,
-      "orderedItems" => items
+      "orderedItems" => if(show_items, do: items, else: [])
     }
 
     if offset < total do

lib/pleroma/web/admin_api/admin_api_controller.ex

@@ -0,0 +1,178 @@
+defmodule Pleroma.Web.AdminAPI.AdminAPIController do
+  use Pleroma.Web, :controller
+  alias Pleroma.{User, Repo}
+  alias Pleroma.Web.ActivityPub.Relay
+
+  import Pleroma.Web.ControllerHelper, only: [json_response: 3]
+
+  require Logger
+
+  action_fallback(:errors)
+
+  def user_delete(conn, %{"nickname" => nickname}) do
+    user = User.get_by_nickname(nickname)
+
+    if user.local == true do
+      User.delete(user)
+    else
+      User.delete(user)
+    end
+
+    conn
+    |> json(nickname)
+  end
+
+  def user_create(
+        conn,
+        %{"nickname" => nickname, "email" => email, "password" => password}
+      ) do
+    new_user = %{
+      nickname: nickname,
+      name: nickname,
+      email: email,
+      password: password,
+      password_confirmation: password,
+      bio: "."
+    }
+
+    User.register_changeset(%User{}, new_user)
+    |> Repo.insert!()
+
+    conn
+    |> json(new_user.nickname)
+  end
+
+  def tag_users(conn, %{"nicknames" => nicknames, "tags" => tags}) do
+    with {:ok, _} <- User.tag(nicknames, tags),
+         do: json_response(conn, :no_content, "")
+  end
+
+  def untag_users(conn, %{"nicknames" => nicknames, "tags" => tags}) do
+    with {:ok, _} <- User.untag(nicknames, tags),
+         do: json_response(conn, :no_content, "")
+  end
+
+  def right_add(conn, %{"permission_group" => permission_group, "nickname" => nickname})
+      when permission_group in ["moderator", "admin"] do
+    user = User.get_by_nickname(nickname)
+
+    info =
+      %{}
+      |> Map.put("is_" <> permission_group, true)
+
+    info_cng = User.Info.admin_api_update(user.info, info)
+
+    cng =
+      user
+      |> Ecto.Changeset.change()
+      |> Ecto.Changeset.put_embed(:info, info_cng)
+
+    {:ok, _user} = User.update_and_set_cache(cng)
+
+    json(conn, info)
+  end
+
+  def right_add(conn, _) do
+    conn
+    |> put_status(404)
+    |> json(%{error: "No such permission_group"})
+  end
+
+  def right_get(conn, %{"nickname" => nickname}) do
+    user = User.get_by_nickname(nickname)
+
+    conn
+    |> json(%{
+      is_moderator: user.info.is_moderator,
+      is_admin: user.info.is_admin
+    })
+  end
+
+  def right_delete(
+        %{assigns: %{user: %User{:nickname => admin_nickname}}} = conn,
+        %{
+          "permission_group" => permission_group,
+          "nickname" => nickname
+        }
+      )
+      when permission_group in ["moderator", "admin"] do
+    if admin_nickname == nickname do
+      conn
+      |> put_status(403)
+      |> json(%{error: "You can't revoke your own admin status."})
+    else
+      user = User.get_by_nickname(nickname)
+
+      info =
+        %{}
+        |> Map.put("is_" <> permission_group, false)
+
+      info_cng = User.Info.admin_api_update(user.info, info)
+
+      cng =
+        Ecto.Changeset.change(user)
+        |> Ecto.Changeset.put_embed(:info, info_cng)
+
+      {:ok, _user} = User.update_and_set_cache(cng)
+
+      json(conn, info)
+    end
+  end
+
+  def right_delete(conn, _) do
+    conn
+    |> put_status(404)
+    |> json(%{error: "No such permission_group"})
+  end
+
+  def relay_follow(conn, %{"relay_url" => target}) do
+    with {:ok, _message} <- Relay.follow(target) do
+      json(conn, target)
+    else
+      _ ->
+        conn
+        |> put_status(500)
+        |> json(target)
+    end
+  end
+
+  def relay_unfollow(conn, %{"relay_url" => target}) do
+    with {:ok, _message} <- Relay.unfollow(target) do
+      json(conn, target)
+    else
+      _ ->
+        conn
+        |> put_status(500)
+        |> json(target)
+    end
+  end
+
+  @doc "Get a account registeration invite token (base64 string)"
+  def get_invite_token(conn, _params) do
+    {:ok, token} = Pleroma.UserInviteToken.create_token()
+
+    conn
+    |> json(token.token)
+  end
+
+  @doc "Get a password reset token (base64 string) for given nickname"
+  def get_password_reset(conn, %{"nickname" => nickname}) do
+    (%User{local: true} = user) = User.get_by_nickname(nickname)
+    {:ok, token} = Pleroma.PasswordResetToken.create_token(user)
+
+    conn
+    |> json(token.token)
+  end
+
+  def errors(conn, {:param_cast, _}) do
+    conn
+    |> put_status(400)
+    |> json("Invalid parameters")
+  end
+
+  def errors(conn, _) do
+    conn
+    |> put_status(500)
+    |> json("Something went wrong")
+  end
+end

lib/pleroma/web/channels/user_socket.ex

@@ -4,9 +4,7 @@ defmodule Pleroma.Web.UserSocket do
 
   ## Channels
   # channel "room:*", Pleroma.Web.RoomChannel
-  if Application.get_env(:pleroma, :chat) |> Keyword.get(:enabled) do
-    channel("chat:*", Pleroma.Web.ChatChannel)
-  end
+  channel("chat:*", Pleroma.Web.ChatChannel)
 
   ## Transports
   transport(:websocket, Phoenix.Transports.WebSocket)
@@ -24,7 +22,8 @@ defmodule Pleroma.Web.UserSocket do
   # See `Phoenix.Token` documentation for examples in
   # performing token verification on connect.
   def connect(%{"token" => token}, socket) do
-    with {:ok, user_id} <- Phoenix.Token.verify(socket, "user socket", token, max_age: 84600),
+    with true <- Pleroma.Config.get([:chat, :enabled]),
+         {:ok, user_id} <- Phoenix.Token.verify(socket, "user socket", token, max_age: 84600),
          %User{} = user <- Pleroma.Repo.get(User, user_id) do
       {:ok, assign(socket, :user_name, user.nickname)}
     else

lib/pleroma/web/common_api/common_api.ex

@@ -8,7 +8,7 @@ defmodule Pleroma.Web.CommonAPI do
   def delete(activity_id, user) do
     with %Activity{data: %{"object" => %{"id" => object_id}}} <- Repo.get(Activity, activity_id),
          %Object{} = object <- Object.normalize(object_id),
-         true <- user.info["is_moderator"] || user.ap_id == object.data["actor"],
+         true <- user.info.is_moderator || user.ap_id == object.data["actor"],
          {:ok, delete} <- ActivityPub.delete(object) do
       {:ok, delete}
     end
@@ -36,7 +36,6 @@ defmodule Pleroma.Web.CommonAPI do
 
   def favorite(id_or_ap_id, user) do
     with %Activity{} = activity <- get_by_id_or_ap_id(id_or_ap_id),
-         false <- activity.data["actor"] == user.ap_id,
          object <- Object.normalize(activity.data["object"]["id"]) do
       ActivityPub.like(user, object)
     else
@@ -47,7 +46,6 @@ defmodule Pleroma.Web.CommonAPI do
 
   def unfavorite(id_or_ap_id, user) do
     with %Activity{} = activity <- get_by_id_or_ap_id(id_or_ap_id),
-         false <- activity.data["actor"] == user.ap_id,
          object <- Object.normalize(activity.data["object"]["id"]) do
       ActivityPub.unlike(user, object)
     else
@@ -72,22 +70,37 @@ defmodule Pleroma.Web.CommonAPI do
 
   def get_visibility(_), do: "public"
 
-  @instance Application.get_env(:pleroma, :instance)
-  @limit Keyword.get(@instance, :limit)
+  defp get_content_type(content_type) do
+    if Enum.member?(Pleroma.Config.get([:instance, :allowed_post_formats]), content_type) do
+      content_type
+    else
+      "text/plain"
+    end
+  end
+
   def post(user, %{"status" => status} = data) do
     visibility = get_visibility(data)
+    limit = Pleroma.Config.get([:instance, :limit])
 
     with status <- String.trim(status),
-         length when length in 1..@limit <- String.length(status),
          attachments <- attachments_from_ids(data["media_ids"]),
          mentions <- Formatter.parse_mentions(status),
          inReplyTo <- get_replied_to_activity(data["in_reply_to_status_id"]),
          {to, cc} <- to_for_user_and_mentions(user, mentions, inReplyTo, visibility),
          tags <- Formatter.parse_tags(status, data),
          content_html <-
-           make_content_html(status, mentions, attachments, tags, data["no_attachment_links"]),
+           make_content_html(
+             status,
+             mentions,
+             attachments,
+             tags,
+             get_content_type(data["content_type"]),
+             data["no_attachment_links"]
+           ),
          context <- make_context(inReplyTo),
          cw <- data["spoiler_text"],
+         full_payload <- String.trim(status <> (data["spoiler_text"] || "")),
+         length when length in 1..limit <- String.length(full_payload),
          object <-
            make_note_data(
              user.ap_id,
@@ -122,12 +135,13 @@ defmodule Pleroma.Web.CommonAPI do
     end
   end
 
+  # Updates the emojis for a user based on their profile
   def update(user) do
     user =
       with emoji <- emoji_from_profile(user),
-           source_data <- (user.info["source_data"] || %{}) |> Map.put("tag", emoji),
-           new_info <- Map.put(user.info, "source_data", source_data),
-           change <- User.info_changeset(user, %{info: new_info}),
+           source_data <- (user.info.source_data || %{}) |> Map.put("tag", emoji),
+           info_cng <- Pleroma.User.Info.set_source_data(user.info, source_data),
+           change <- Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_cng),
            {:ok, user} <- User.update_and_set_cache(change) do
         user
       else

lib/pleroma/web/common_api/utils.ex

@@ -2,6 +2,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   alias Pleroma.{Repo, Object, Formatter, Activity}
   alias Pleroma.Web.ActivityPub.Utils
   alias Pleroma.Web.Endpoint
+  alias Pleroma.Web.MediaProxy
   alias Pleroma.User
   alias Calendar.Strftime
   alias Comeonin.Pbkdf2
@@ -18,6 +19,8 @@ defmodule Pleroma.Web.CommonAPI.Utils do
       end
   end
 
+  def get_replied_to_activity(""), do: nil
+
   def get_replied_to_activity(id) when not is_nil(id) do
     Repo.get(Activity, id)
   end
@@ -31,21 +34,29 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   end
 
   def to_for_user_and_mentions(user, mentions, inReplyTo, "public") do
-    to = ["https://www.w3.org/ns/activitystreams#Public"]
-
     mentioned_users = Enum.map(mentions, fn {_, %{ap_id: ap_id}} -> ap_id end)
-    cc = [user.follower_address | mentioned_users]
+
+    to = ["https://www.w3.org/ns/activitystreams#Public" | mentioned_users]
+    cc = [user.follower_address]
 
     if inReplyTo do
-      {to, Enum.uniq([inReplyTo.data["actor"] | cc])}
+      {Enum.uniq([inReplyTo.data["actor"] | to]), cc}
     else
       {to, cc}
     end
   end
 
   def to_for_user_and_mentions(user, mentions, inReplyTo, "unlisted") do
-    {to, cc} = to_for_user_and_mentions(user, mentions, inReplyTo, "public")
-    {cc, to}
+    mentioned_users = Enum.map(mentions, fn {_, %{ap_id: ap_id}} -> ap_id end)
+
+    to = [user.follower_address | mentioned_users]
+    cc = ["https://www.w3.org/ns/activitystreams#Public"]
+
+    if inReplyTo do
+      {Enum.uniq([inReplyTo.data["actor"] | to]), cc}
+    else
+      {to, cc}
+    end
   end
 
   def to_for_user_and_mentions(user, mentions, inReplyTo, "private") do
@@ -63,9 +74,16 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     end
   end
 
-  def make_content_html(status, mentions, attachments, tags, no_attachment_links \\ false) do
+  def make_content_html(
+        status,
+        mentions,
+        attachments,
+        tags,
+        content_type,
+        no_attachment_links \\ false
+      ) do
     status
-    |> format_input(mentions, tags)
+    |> format_input(mentions, tags, content_type)
     |> maybe_add_attachments(attachments, no_attachment_links)
   end
 
@@ -81,8 +99,9 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   def add_attachments(text, attachments) do
     attachment_text =
       Enum.map(attachments, fn
-        %{"url" => [%{"href" => href} | _]} ->
-          name = URI.decode(Path.basename(href))
+        %{"url" => [%{"href" => href} | _]} = attachment ->
+          name = attachment["name"] || URI.decode(Path.basename(href))
+          href = MediaProxy.url(href)
           "<a href=\"#{href}\" class='attachment'>#{shortname(name)}</a>"
 
         _ ->
@@ -92,9 +111,9 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     Enum.join([text | attachment_text], "<br>")
   end
 
-  def format_input(text, mentions, tags) do
+  def format_input(text, mentions, tags, "text/plain") do
     text
-    |> Formatter.html_escape()
+    |> Formatter.html_escape("text/plain")
     |> String.replace(~r/\r?\n/, "<br>")
     |> (&{[], &1}).()
     |> Formatter.add_links()
@@ -103,6 +122,26 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     |> Formatter.finalize()
   end
 
+  def format_input(text, mentions, _tags, "text/html") do
+    text
+    |> Formatter.html_escape("text/html")
+    |> String.replace(~r/\r?\n/, "<br>")
+    |> (&{[], &1}).()
+    |> Formatter.add_user_links(mentions)
+    |> Formatter.finalize()
+  end
+
+  def format_input(text, mentions, tags, "text/markdown") do
+    text
+    |> Earmark.as_html!()
+    |> Formatter.html_escape("text/html")
+    |> String.replace(~r/\r?\n/, "")
+    |> (&{[], &1}).()
+    |> Formatter.add_user_links(mentions)
+    |> Formatter.add_hashtag_links(tags)
+    |> Formatter.finalize()
+  end
+
   def add_tag_links(text, tags) do
     tags =
       tags
@@ -197,7 +236,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     end
   end
 
-  def emoji_from_profile(%{info: info} = user) do
+  def emoji_from_profile(%{info: _info} = user) do
     (Formatter.get_emoji(user.bio) ++ Formatter.get_emoji(user.name))
     |> Enum.map(fn {shortcode, url} ->
       %{

lib/pleroma/web/controller_helper.ex

@@ -0,0 +1,9 @@
+defmodule Pleroma.Web.ControllerHelper do
+  use Pleroma.Web, :controller
+
+  def json_response(conn, status, json) do
+    conn
+    |> put_status(status)
+    |> json(json)
+  end
+end

lib/pleroma/web/endpoint.ex

@@ -1,9 +1,7 @@
 defmodule Pleroma.Web.Endpoint do
   use Phoenix.Endpoint, otp_app: :pleroma
 
-  if Application.get_env(:pleroma, :chat) |> Keyword.get(:enabled) do
-    socket("/socket", Pleroma.Web.UserSocket)
-  end
+  socket("/socket", Pleroma.Web.UserSocket)
 
   socket("/api/v1", Pleroma.Web.MastodonAPI.MastodonSocket)
 
@@ -11,13 +9,17 @@ defmodule Pleroma.Web.Endpoint do
   #
   # You should set gzip to true if you are running phoenix.digest
   # when deploying your static files in production.
-  plug(Plug.Static, at: "/media", from: Pleroma.Uploaders.Local.upload_path(), gzip: false)
+  plug(CORSPlug)
+  plug(Pleroma.Plugs.HTTPSecurityPlug)
+
+  plug(Pleroma.Plugs.UploadedMedia)
 
   plug(
     Plug.Static,
     at: "/",
     from: :pleroma,
-    only: ~w(index.html static finmoji emoji packs sounds images instance sw.js favicon.png)
+    only:
+      ~w(index.html static finmoji emoji packs sounds images instance sw.js favicon.png schemas)
   )
 
   # Code reloading can be explicitly enabled under the
@@ -42,14 +44,19 @@ defmodule Pleroma.Web.Endpoint do
   plug(Plug.MethodOverride)
   plug(Plug.Head)
 
+  cookie_name =
+    if Application.get_env(:pleroma, Pleroma.Web.Endpoint) |> Keyword.get(:secure_cookie_flag),
+      do: "__Host-pleroma_key",
+      else: "pleroma_key"
+
   # The session will be stored in the cookie and signed,
   # this means its contents can be read but not tampered with.
   # Set :encryption_salt if you would also like to encrypt it.
   plug(
     Plug.Session,
     store: :cookie,
-    key: "_pleroma_key",
-    signing_salt: "CqaoopA2",
+    key: cookie_name,
+    signing_salt: {Pleroma.Config, :get, [[__MODULE__, :signing_salt], "CqaoopA2"]},
     http_only: true,
     secure:
       Application.get_env(:pleroma, Pleroma.Web.Endpoint) |> Keyword.get(:secure_cookie_flag),

lib/pleroma/web/federator/federator.ex

@@ -3,17 +3,16 @@ defmodule Pleroma.Web.Federator do
   alias Pleroma.User
   alias Pleroma.Activity
   alias Pleroma.Web.{WebFinger, Websub}
+  alias Pleroma.Web.Federator.RetryQueue
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Relay
   alias Pleroma.Web.ActivityPub.Transmogrifier
   alias Pleroma.Web.ActivityPub.Utils
+  alias Pleroma.Web.OStatus
   require Logger
 
   @websub Application.get_env(:pleroma, :websub)
   @ostatus Application.get_env(:pleroma, :ostatus)
-  @httpoison Application.get_env(:pleroma, :httpoison)
-  @instance Application.get_env(:pleroma, :instance)
-  @federating Keyword.get(@instance, :federating)
   @max_jobs 20
 
   def init(args) do
@@ -65,15 +64,17 @@ defmodule Pleroma.Web.Federator do
       {:ok, actor} = WebFinger.ensure_keys_present(actor)
 
       if ActivityPub.is_public?(activity) do
-        Logger.info(fn -> "Sending #{activity.data["id"]} out via WebSub" end)
-        Websub.publish(Pleroma.Web.OStatus.feed_path(actor), actor, activity)
+        if OStatus.is_representable?(activity) do
+          Logger.info(fn -> "Sending #{activity.data["id"]} out via WebSub" end)
+          Websub.publish(Pleroma.Web.OStatus.feed_path(actor), actor, activity)
 
-        Logger.info(fn -> "Sending #{activity.data["id"]} out via Salmon" end)
-        Pleroma.Web.Salmon.publish(actor, activity)
+          Logger.info(fn -> "Sending #{activity.data["id"]} out via Salmon" end)
+          Pleroma.Web.Salmon.publish(actor, activity)
+        end
 
-        if Mix.env() != :test do
+        if Keyword.get(Application.get_env(:pleroma, :instance), :allow_relay) do
           Logger.info(fn -> "Relaying #{activity.data["id"]} out" end)
-          Pleroma.Web.ActivityPub.Relay.publish(activity)
+          Relay.publish(activity)
         end
       end
 
@@ -100,44 +101,46 @@ defmodule Pleroma.Web.Federator do
 
     params = Utils.normalize_params(params)
 
+    # NOTE: we use the actor ID to do the containment, this is fine because an
+    # actor shouldn't be acting on objects outside their own AP server.
     with {:ok, _user} <- ap_enabled_actor(params["actor"]),
          nil <- Activity.normalize(params["id"]),
-         {:ok, _activity} <- Transmogrifier.handle_incoming(params) do
+         :ok <- Transmogrifier.contain_origin_from_id(params["actor"], params),
+         {:ok, activity} <- Transmogrifier.handle_incoming(params) do
+      {:ok, activity}
     else
       %Activity{} ->
         Logger.info("Already had #{params["id"]}")
+        :error
 
       _e ->
         # Just drop those for now
         Logger.info("Unhandled activity")
         Logger.info(Poison.encode!(params, pretty: 2))
+        :error
     end
   end
 
   def handle(:publish_single_ap, params) do
-    ActivityPub.publish_one(params)
+    case ActivityPub.publish_one(params) do
+      {:ok, _} ->
+        :ok
+
+      {:error, _} ->
+        RetryQueue.enqueue(params, ActivityPub)
+    end
   end
 
-  def handle(:publish_single_websub, %{xml: xml, topic: topic, callback: callback, secret: secret}) do
-    signature = @websub.sign(secret || "", xml)
-    Logger.debug(fn -> "Pushing #{topic} to #{callback}" end)
+  def handle(
+        :publish_single_websub,
+        %{xml: _xml, topic: _topic, callback: _callback, secret: _secret} = params
+      ) do
+    case Websub.publish_one(params) do
+      {:ok, _} ->
+        :ok
 
-    with {:ok, %{status_code: code}} <-
-           @httpoison.post(
-             callback,
-             xml,
-             [
-               {"Content-Type", "application/atom+xml"},
-               {"X-Hub-Signature", "sha1=#{signature}"}
-             ],
-             timeout: 10000,
-             recv_timeout: 20000,
-             hackney: [pool: :default]
-           ) do
-      Logger.debug(fn -> "Pushed to #{callback}, code #{code}" end)
-    else
-      e ->
-        Logger.debug(fn -> "Couldn't push to #{callback}, #{inspect(e)}" end)
+      {:error, _} ->
+        RetryQueue.enqueue(params, Websub)
     end
   end
 
@@ -146,11 +149,15 @@ defmodule Pleroma.Web.Federator do
     {:error, "Don't know what to do with this"}
   end
 
-  def enqueue(type, payload, priority \\ 1) do
-    if @federating do
-      if Mix.env() == :test do
+  if Mix.env() == :test do
+    def enqueue(type, payload, _priority \\ 1) do
+      if Pleroma.Config.get([:instance, :federating]) do
         handle(type, payload)
-      else
+      end
+    end
+  else
+    def enqueue(type, payload, priority \\ 1) do
+      if Pleroma.Config.get([:instance, :federating]) do
         GenServer.cast(__MODULE__, {:enqueue, type, payload, priority})
       end
     end

lib/pleroma/web/federator/retry_queue.ex

@@ -0,0 +1,74 @@
+defmodule Pleroma.Web.Federator.RetryQueue do
+  use GenServer
+
+  require Logger
+
+  # initial timeout, 5 min
+  @initial_timeout 30_000
+  @max_retries 5
+
+  def init(args) do
+    {:ok, args}
+  end
+
+  def start_link() do
+    enabled = Pleroma.Config.get([:retry_queue, :enabled], false)
+
+    if enabled do
+      Logger.info("Starting retry queue")
+      GenServer.start_link(__MODULE__, %{delivered: 0, dropped: 0}, name: __MODULE__)
+    else
+      Logger.info("Retry queue disabled")
+      :ignore
+    end
+  end
+
+  def enqueue(data, transport, retries \\ 0) do
+    GenServer.cast(__MODULE__, {:maybe_enqueue, data, transport, retries + 1})
+  end
+
+  def get_retry_params(retries) do
+    if retries > @max_retries do
+      {:drop, "Max retries reached"}
+    else
+      {:retry, growth_function(retries)}
+    end
+  end
+
+  def handle_cast({:maybe_enqueue, data, transport, retries}, %{dropped: drop_count} = state) do
+    case get_retry_params(retries) do
+      {:retry, timeout} ->
+        Process.send_after(
+          __MODULE__,
+          {:send, data, transport, retries},
+          timeout
+        )
+
+        {:noreply, state}
+
+      {:drop, message} ->
+        Logger.debug(message)
+        {:noreply, %{state | dropped: drop_count + 1}}
+    end
+  end
+
+  def handle_info({:send, data, transport, retries}, %{delivered: delivery_count} = state) do
+    case transport.publish_one(data) do
+      {:ok, _} ->
+        {:noreply, %{state | delivered: delivery_count + 1}}
+
+      {:error, _reason} ->
+        enqueue(data, transport, retries)
+        {:noreply, state}
+    end
+  end
+
+  def handle_info(unknown, state) do
+    Logger.debug("RetryQueue: don't know what to do with #{inspect(unknown)}, ignoring")
+    {:noreply, state}
+  end
+
+  defp growth_function(retries) do
+    round(@initial_timeout * :math.pow(retries, 3))
+  end
+end

lib/pleroma/web/http_signatures/http_signatures.ex

@@ -65,7 +65,7 @@ defmodule Pleroma.Web.HTTPSignatures do
   end
 
   def sign(user, headers) do
-    with {:ok, %{info: %{"keys" => keys}}} <- Pleroma.Web.WebFinger.ensure_keys_present(user),
+    with {:ok, %{info: %{keys: keys}}} <- Pleroma.Web.WebFinger.ensure_keys_present(user),
          {:ok, private_key, _} = Pleroma.Web.Salmon.keys_from_pem(keys) do
       sigstring = build_signing_string(headers, Map.keys(headers))
 

lib/pleroma/web/mastodon_api/mastodon_api_controller.ex

@@ -2,13 +2,22 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   use Pleroma.Web, :controller
   alias Pleroma.{Repo, Object, Activity, User, Notification, Stats}
   alias Pleroma.Web
-  alias Pleroma.Web.MastodonAPI.{StatusView, AccountView, MastodonView, ListView, FilterView}
+
+  alias Pleroma.Web.MastodonAPI.{
+    StatusView,
+    AccountView,
+    MastodonView,
+    ListView,
+    FilterView,
+    PushSubscriptionView
+  }
+
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Utils
   alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.OAuth.{Authorization, Token, App}
   alias Pleroma.Web.MediaProxy
-  alias Comeonin.Pbkdf2
+
   import Ecto.Query
   require Logger
 
@@ -32,73 +41,61 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
+  defp add_if_present(
+         map,
+         params,
+         params_field,
+         map_field,
+         value_function \\ fn x -> {:ok, x} end
+       ) do
+    if Map.has_key?(params, params_field) do
+      case value_function.(params[params_field]) do
+        {:ok, new_value} -> Map.put(map, map_field, new_value)
+        :error -> map
+      end
+    else
+      map
+    end
+  end
+
   def update_credentials(%{assigns: %{user: user}} = conn, params) do
     original_user = user
 
-    params =
-      if bio = params["note"] do
-        Map.put(params, "bio", bio)
-      else
-        params
-      end
-
-    params =
-      if name = params["display_name"] do
-        Map.put(params, "name", name)
-      else
-        params
-      end
-
-    user =
-      if avatar = params["avatar"] do
-        with %Plug.Upload{} <- avatar,
-             {:ok, object} <- ActivityPub.upload(avatar),
-             change = Ecto.Changeset.change(user, %{avatar: object.data}),
-             {:ok, user} = User.update_and_set_cache(change) do
-          user
+    user_params =
+      %{}
+      |> add_if_present(params, "display_name", :name)
+      |> add_if_present(params, "note", :bio, fn value -> {:ok, User.parse_bio(value)} end)
+      |> add_if_present(params, "avatar", :avatar, fn value ->
+        with %Plug.Upload{} <- value,
+             {:ok, object} <- ActivityPub.upload(value, type: :avatar) do
+          {:ok, object.data}
         else
-          _e -> user
+          _ -> :error
         end
-      else
-        user
-      end
+      end)
 
-    user =
-      if banner = params["header"] do
-        with %Plug.Upload{} <- banner,
-             {:ok, object} <- ActivityPub.upload(banner),
-             new_info <- Map.put(user.info, "banner", object.data),
-             change <- User.info_changeset(user, %{info: new_info}),
-             {:ok, user} <- User.update_and_set_cache(change) do
-          user
+    info_params =
+      %{}
+      |> add_if_present(params, "locked", :locked, fn value -> {:ok, value == "true"} end)
+      |> add_if_present(params, "header", :banner, fn value ->
+        with %Plug.Upload{} <- value,
+             {:ok, object} <- ActivityPub.upload(value, type: :banner) do
+          {:ok, object.data}
         else
-          _e -> user
+          _ -> :error
         end
-      else
-        user
-      end
+      end)
 
-    user =
-      if locked = params["locked"] do
-        with locked <- locked == "true",
-             new_info <- Map.put(user.info, "locked", locked),
-             change <- User.info_changeset(user, %{info: new_info}),
-             {:ok, user} <- User.update_and_set_cache(change) do
-          user
-        else
-          _e -> user
-        end
-      else
-        user
-      end
+    info_cng = User.Info.mastodon_profile_update(user.info, info_params)
 
-    with changeset <- User.update_changeset(user, params),
+    with changeset <- User.update_changeset(user, user_params),
+         changeset <- Ecto.Changeset.put_embed(changeset, :info, info_cng),
          {:ok, user} <- User.update_and_set_cache(changeset) do
       if original_user != user do
         CommonAPI.update(user)
       end
 
-      json(conn, AccountView.render("account.json", %{user: user}))
+      json(conn, AccountView.render("account.json", %{user: user, for: user}))
     else
       _e ->
         conn
@@ -108,13 +105,13 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   end
 
   def verify_credentials(%{assigns: %{user: user}} = conn, _) do
-    account = AccountView.render("account.json", %{user: user})
+    account = AccountView.render("account.json", %{user: user, for: user})
     json(conn, account)
   end
 
-  def user(conn, %{"id" => id}) do
+  def user(%{assigns: %{user: for_user}} = conn, %{"id" => id}) do
     with %User{} = user <- Repo.get(User, id) do
-      account = AccountView.render("account.json", %{user: user})
+      account = AccountView.render("account.json", %{user: user, for: for_user})
       json(conn, account)
     else
       _e ->
@@ -124,22 +121,23 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
-  @instance Application.get_env(:pleroma, :instance)
-  @mastodon_api_level "2.4.3"
+  @mastodon_api_level "2.5.0"
 
   def masto_instance(conn, _params) do
+    instance = Pleroma.Config.get(:instance)
+
     response = %{
       uri: Web.base_url(),
-      title: Keyword.get(@instance, :name),
-      description: Keyword.get(@instance, :description),
-      version: "#{@mastodon_api_level} (compatible; #{Keyword.get(@instance, :version)})",
-      email: Keyword.get(@instance, :email),
+      title: Keyword.get(instance, :name),
+      description: Keyword.get(instance, :description),
+      version: "#{@mastodon_api_level} (compatible; #{Pleroma.Application.named_version()})",
+      email: Keyword.get(instance, :email),
       urls: %{
         streaming_api: String.replace(Pleroma.Web.Endpoint.static_url(), "http", "ws")
       },
       stats: Stats.get_stats(),
       thumbnail: Web.base_url() <> "/instance/thumbnail.jpeg",
-      max_toot_chars: Keyword.get(@instance, :limit)
+      max_toot_chars: Keyword.get(instance, :limit)
     }
 
     json(conn, response)
@@ -150,7 +148,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   end
 
   defp mastodonized_emoji do
-    Pleroma.Formatter.get_custom_emoji()
+    Pleroma.Emoji.get_all()
     |> Enum.map(fn {shortcode, relative_url} ->
       url = to_string(URI.merge(Web.base_url(), relative_url))
 
@@ -223,6 +221,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
 
     activities =
       ActivityPub.fetch_activities([user.ap_id | user.following], params)
+      |> ActivityPub.contain_timeline(user)
       |> Enum.reverse()
 
     conn
@@ -268,9 +267,12 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
-  def dm_timeline(%{assigns: %{user: user}} = conn, _params) do
+  def dm_timeline(%{assigns: %{user: user}} = conn, params) do
     query =
-      ActivityPub.fetch_activities_query([user.ap_id], %{"type" => "Create", visibility: "direct"})
+      ActivityPub.fetch_activities_query(
+        [user.ap_id],
+        Map.merge(params, %{"type" => "Create", visibility: "direct"})
+      )
 
     activities = Repo.all(query)
 
@@ -282,7 +284,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   def get_status(%{assigns: %{user: user}} = conn, %{"id" => id}) do
     with %Activity{} = activity <- Repo.get(Activity, id),
          true <- ActivityPub.visible_for_user?(activity, user) do
-      render(conn, StatusView, "status.json", %{activity: activity, for: user})
+      try_render(conn, StatusView, "status.json", %{activity: activity, for: user})
     end
   end
 
@@ -345,7 +347,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     {:ok, activity} =
       Cachex.fetch!(:idempotency_cache, idempotency_key, fn _ -> CommonAPI.post(user, params) end)
 
-    render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
+    try_render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
   end
 
   def delete_status(%{assigns: %{user: user}} = conn, %{"id" => id}) do
@@ -361,28 +363,28 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
 
   def reblog_status(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
     with {:ok, announce, _activity} <- CommonAPI.repeat(ap_id_or_id, user) do
-      render(conn, StatusView, "status.json", %{activity: announce, for: user, as: :activity})
+      try_render(conn, StatusView, "status.json", %{activity: announce, for: user, as: :activity})
     end
   end
 
   def unreblog_status(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
     with {:ok, _unannounce, %{data: %{"id" => id}}} <- CommonAPI.unrepeat(ap_id_or_id, user),
          %Activity{} = activity <- Activity.get_create_activity_by_object_ap_id(id) do
-      render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
+      try_render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
     end
   end
 
   def fav_status(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
     with {:ok, _fav, %{data: %{"id" => id}}} <- CommonAPI.favorite(ap_id_or_id, user),
          %Activity{} = activity <- Activity.get_create_activity_by_object_ap_id(id) do
-      render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
+      try_render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
     end
   end
 
   def unfav_status(%{assigns: %{user: user}} = conn, %{"id" => ap_id_or_id}) do
     with {:ok, _, _, %{data: %{"id" => id}}} <- CommonAPI.unfavorite(ap_id_or_id, user),
          %Activity{} = activity <- Activity.get_create_activity_by_object_ap_id(id) do
-      render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
+      try_render(conn, StatusView, "status.json", %{activity: activity, for: user, as: :activity})
     end
   end
 
@@ -434,40 +436,34 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     render(conn, AccountView, "relationships.json", %{user: user, targets: targets})
   end
 
-  def update_media(%{assigns: %{user: _}} = conn, data) do
+  # Instead of returning a 400 when no "id" params is present, Mastodon returns an empty array.
+  def relationships(%{assigns: %{user: _user}} = conn, _), do: json(conn, [])
+
+  def update_media(%{assigns: %{user: user}} = conn, data) do
     with %Object{} = object <- Repo.get(Object, data["id"]),
+         true <- Object.authorize_mutation(object, user),
          true <- is_binary(data["description"]),
          description <- data["description"] do
       new_data = %{object.data | "name" => description}
 
-      change = Object.change(object, %{data: new_data})
-      {:ok, media_obj} = Repo.update(change)
+      {:ok, _} =
+        object
+        |> Object.change(%{data: new_data})
+        |> Repo.update()
 
-      data =
-        new_data
-        |> Map.put("id", object.id)
-
-      render(conn, StatusView, "attachment.json", %{attachment: data})
+      attachment_data = Map.put(new_data, "id", object.id)
+      render(conn, StatusView, "attachment.json", %{attachment: attachment_data})
     end
   end
 
-  def upload(%{assigns: %{user: _}} = conn, %{"file" => file} = data) do
-    with {:ok, object} <- ActivityPub.upload(file) do
-      objdata =
-        if Map.has_key?(data, "description") do
-          Map.put(object.data, "name", data["description"])
-        else
-          object.data
-        end
-
-      change = Object.change(object, %{data: objdata})
-      {:ok, object} = Repo.update(change)
-
-      objdata =
-        objdata
-        |> Map.put("id", object.id)
-
-      render(conn, StatusView, "attachment.json", %{attachment: objdata})
+  def upload(%{assigns: %{user: user}} = conn, %{"file" => file} = data) do
+    with {:ok, object} <-
+           ActivityPub.upload(file,
+             actor: User.ap_id(user),
+             description: Map.get(data, "description")
+           ) do
+      attachment_data = Map.put(object.data, "id", object.id)
+      render(conn, StatusView, "attachment.json", %{attachment: attachment_data})
     end
   end
 
@@ -499,6 +495,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
       |> Map.put("type", "Create")
       |> Map.put("local_only", local_only)
       |> Map.put("blocking_user", user)
+      |> Map.put("tag", String.downcase(params["tag"]))
 
     activities =
       ActivityPub.fetch_public_activities(params)
@@ -509,17 +506,30 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     |> render(StatusView, "index.json", %{activities: activities, for: user, as: :activity})
   end
 
-  # TODO: Pagination
-  def followers(conn, %{"id" => id}) do
+  def followers(%{assigns: %{user: for_user}} = conn, %{"id" => id}) do
     with %User{} = user <- Repo.get(User, id),
          {:ok, followers} <- User.get_followers(user) do
+      followers =
+        cond do
+          for_user && user.id == for_user.id -> followers
+          user.info.hide_network -> []
+          true -> followers
+        end
+
       render(conn, AccountView, "accounts.json", %{users: followers, as: :user})
     end
   end
 
-  def following(conn, %{"id" => id}) do
+  def following(%{assigns: %{user: for_user}} = conn, %{"id" => id}) do
     with %User{} = user <- Repo.get(User, id),
          {:ok, followers} <- User.get_friends(user) do
+      followers =
+        cond do
+          for_user && user.id == for_user.id -> followers
+          user.info.hide_network -> []
+          true -> followers
+        end
+
       render(conn, AccountView, "accounts.json", %{users: followers, as: :user})
     end
   end
@@ -574,7 +584,13 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   def follow(%{assigns: %{user: follower}} = conn, %{"id" => id}) do
     with %User{} = followed <- Repo.get(User, id),
          {:ok, follower} <- User.maybe_direct_follow(follower, followed),
-         {:ok, _activity} <- ActivityPub.follow(follower, followed) do
+         {:ok, _activity} <- ActivityPub.follow(follower, followed),
+         {:ok, follower, followed} <-
+           User.wait_and_refresh(
+             Pleroma.Config.get([:activitypub, :follow_handshake_timeout]),
+             follower,
+             followed
+           ) do
       render(conn, AccountView, "relationship.json", %{user: follower, target: followed})
     else
       {:error, message} ->
@@ -588,7 +604,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     with %User{} = followed <- Repo.get_by(User, nickname: uri),
          {:ok, follower} <- User.maybe_direct_follow(follower, followed),
          {:ok, _activity} <- ActivityPub.follow(follower, followed) do
-      render(conn, AccountView, "account.json", %{user: followed})
+      render(conn, AccountView, "account.json", %{user: followed, for: follower})
     else
       {:error, message} ->
         conn
@@ -633,7 +649,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
 
   # TODO: Use proper query
   def blocks(%{assigns: %{user: user}} = conn, _) do
-    with blocked_users <- user.info["blocks"] || [],
+    with blocked_users <- user.info.blocks || [],
          accounts <- Enum.map(blocked_users, fn ap_id -> User.get_cached_by_ap_id(ap_id) end) do
       res = AccountView.render("accounts.json", users: accounts, for: user, as: :user)
       json(conn, res)
@@ -641,7 +657,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   end
 
   def domain_blocks(%{assigns: %{user: %{info: info}}} = conn, _) do
-    json(conn, info["domain_blocks"] || [])
+    json(conn, info.domain_blocks || [])
   end
 
   def block_domain(%{assigns: %{user: blocker}} = conn, %{"domain" => domain}) do
@@ -765,6 +781,12 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
+  def account_lists(%{assigns: %{user: user}} = conn, %{"id" => account_id}) do
+    lists = Pleroma.List.get_lists_account_belongs(user, account_id)
+    res = ListView.render("lists.json", lists: lists)
+    json(conn, res)
+  end
+
   def delete_list(%{assigns: %{user: user}} = conn, %{"id" => id}) do
     with %Pleroma.List{} = list <- Pleroma.List.get(id, user),
          {:ok, _list} <- Pleroma.List.delete(list) do
@@ -825,7 +847,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   end
 
   def list_timeline(%{assigns: %{user: user}} = conn, %{"list_id" => id} = params) do
-    with %Pleroma.List{title: title, following: following} <- Pleroma.List.get(id, user) do
+    with %Pleroma.List{title: _title, following: following} <- Pleroma.List.get(id, user) do
       params =
         params
         |> Map.put("type", "Create")
@@ -858,7 +880,11 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
 
     if user && token do
       mastodon_emoji = mastodonized_emoji()
-      accounts = Map.put(%{}, user.id, AccountView.render("account.json", %{user: user}))
+
+      limit = Pleroma.Config.get([:instance, :limit])
+
+      accounts =
+        Map.put(%{}, user.id, AccountView.render("account.json", %{user: user, for: user}))
 
       initial_state =
         %{
@@ -876,14 +902,14 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
             auto_play_gif: false,
             display_sensitive_media: false,
             reduce_motion: false,
-            max_toot_chars: Keyword.get(@instance, :limit)
+            max_toot_chars: limit
           },
           rights: %{
-            delete_others_notice: !!user.info["is_moderator"]
+            delete_others_notice: !!user.info.is_moderator
           },
           compose: %{
             me: "#{user.id}",
-            default_privacy: user.info["default_scope"] || "public",
+            default_privacy: user.info.default_scope,
             default_sensitive: false
           },
           media_attachments: %{
@@ -903,7 +929,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
             ]
           },
           settings:
-            Map.get(user.info, "settings") ||
+            Map.get(user.info, :settings) ||
               %{
                 onboarded: true,
                 home: %{
@@ -936,7 +962,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
           push_subscription: nil,
           accounts: accounts,
           custom_emojis: mastodon_emoji,
-          char_limit: Keyword.get(@instance, :limit)
+          char_limit: limit
         }
         |> Jason.encode!()
 
@@ -950,21 +976,41 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   end
 
   def put_settings(%{assigns: %{user: user}} = conn, %{"data" => settings} = _params) do
-    with new_info <- Map.put(user.info, "settings", settings),
-         change <- User.info_changeset(user, %{info: new_info}),
-         {:ok, _user} <- User.update_and_set_cache(change) do
-      conn
-      |> json(%{})
+    info_cng = User.Info.mastodon_settings_update(user.info, settings)
+
+    with changeset <- User.update_changeset(user),
+         changeset <- Ecto.Changeset.put_embed(changeset, :info, info_cng),
+         {:ok, _user} <- User.update_and_set_cache(changeset) do
+      json(conn, %{})
     else
       e ->
-        conn
-        |> json(%{error: inspect(e)})
+        json(conn, %{error: inspect(e)})
+    end
+  end
+
+  def login(conn, %{"code" => code}) do
+    with {:ok, app} <- get_or_make_app(),
+         %Authorization{} = auth <- Repo.get_by(Authorization, token: code, app_id: app.id),
+         {:ok, token} <- Token.exchange_token(app, auth) do
+      conn
+      |> put_session(:oauth_token, token.token)
+      |> redirect(to: "/web/getting-started")
     end
   end
 
   def login(conn, _) do
-    conn
-    |> render(MastodonView, "login.html", %{error: false})
+    with {:ok, app} <- get_or_make_app() do
+      path =
+        o_auth_path(conn, :authorize,
+          response_type: "code",
+          client_id: app.client_id,
+          redirect_uri: ".",
+          scope: app.scopes
+        )
+
+      conn
+      |> redirect(to: path)
+    end
   end
 
   defp get_or_make_app() do
@@ -983,22 +1029,6 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
-  def login_post(conn, %{"authorization" => %{"name" => name, "password" => password}}) do
-    with %User{} = user <- User.get_by_nickname_or_email(name),
-         true <- Pbkdf2.checkpw(password, user.password_hash),
-         {:ok, app} <- get_or_make_app(),
-         {:ok, auth} <- Authorization.create_authorization(app, user),
-         {:ok, token} <- Token.exchange_token(app, auth) do
-      conn
-      |> put_session(:oauth_token, token.token)
-      |> redirect(to: "/web/getting-started")
-    else
-      _e ->
-        conn
-        |> render(MastodonView, "login.html", %{error: "Wrong username or password"})
-    end
-  end
-
   def logout(conn, _) do
     conn
     |> clear_session
@@ -1038,7 +1068,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
           id: id,
           type: "mention",
           created_at: created_at,
-          account: AccountView.render("account.json", %{user: actor}),
+          account: AccountView.render("account.json", %{user: actor, for: user}),
           status: StatusView.render("status.json", %{activity: activity, for: user})
         }
 
@@ -1049,7 +1079,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
           id: id,
           type: "favourite",
           created_at: created_at,
-          account: AccountView.render("account.json", %{user: actor}),
+          account: AccountView.render("account.json", %{user: actor, for: user}),
           status: StatusView.render("status.json", %{activity: liked_activity, for: user})
         }
 
@@ -1060,7 +1090,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
           id: id,
           type: "reblog",
           created_at: created_at,
-          account: AccountView.render("account.json", %{user: actor}),
+          account: AccountView.render("account.json", %{user: actor, for: user}),
           status: StatusView.render("status.json", %{activity: announced_activity, for: user})
         }
 
@@ -1069,7 +1099,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
           id: id,
           type: "follow",
           created_at: created_at,
-          account: AccountView.render("account.json", %{user: actor})
+          account: AccountView.render("account.json", %{user: actor, for: user})
         }
 
       _ ->
@@ -1077,7 +1107,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
-  def get_filters(%{assigns: %{user: user}} = conn, params) do
+  def get_filters(%{assigns: %{user: user}} = conn, _) do
     filters = Pleroma.Filter.get_filters(user)
     res = FilterView.render("filters.json", filters: filters)
     json(conn, res)
@@ -1101,7 +1131,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     json(conn, res)
   end
 
-  def get_filter(%{assigns: %{user: user}} = conn, %{"id" => filter_id} = params) do
+  def get_filter(%{assigns: %{user: user}} = conn, %{"id" => filter_id}) do
     filter = Pleroma.Filter.get(filter_id, user)
     res = FilterView.render("filter.json", filter: filter)
     json(conn, res)
@@ -1126,13 +1156,40 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     json(conn, res)
   end
 
-  def delete_filter(%{assigns: %{user: user}} = conn, %{"id" => filter_id} = params) do
+  def delete_filter(%{assigns: %{user: user}} = conn, %{"id" => filter_id}) do
     query = %Pleroma.Filter{
       user_id: user.id,
       filter_id: filter_id
     }
 
-    {:ok, response} = Pleroma.Filter.delete(query)
+    {:ok, _} = Pleroma.Filter.delete(query)
+    json(conn, %{})
+  end
+
+  def create_push_subscription(%{assigns: %{user: user, token: token}} = conn, params) do
+    Pleroma.Web.Push.Subscription.delete_if_exists(user, token)
+    {:ok, subscription} = Pleroma.Web.Push.Subscription.create(user, token, params)
+    view = PushSubscriptionView.render("push_subscription.json", subscription: subscription)
+    json(conn, view)
+  end
+
+  def get_push_subscription(%{assigns: %{user: user, token: token}} = conn, _params) do
+    subscription = Pleroma.Web.Push.Subscription.get(user, token)
+    view = PushSubscriptionView.render("push_subscription.json", subscription: subscription)
+    json(conn, view)
+  end
+
+  def update_push_subscription(
+        %{assigns: %{user: user, token: token}} = conn,
+        params
+      ) do
+    {:ok, subscription} = Pleroma.Web.Push.Subscription.update(user, token, params)
+    view = PushSubscriptionView.render("push_subscription.json", subscription: subscription)
+    json(conn, view)
+  end
+
+  def delete_push_subscription(%{assigns: %{user: user, token: token}} = conn, _params) do
+    {:ok, _response} = Pleroma.Web.Push.Subscription.delete(user, token)
     json(conn, %{})
   end
 
@@ -1142,24 +1199,28 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     |> json("Something went wrong")
   end
 
-  @suggestions Application.get_env(:pleroma, :suggestions)
-
   def suggestions(%{assigns: %{user: user}} = conn, _) do
-    if Keyword.get(@suggestions, :enabled, false) do
-      api = Keyword.get(@suggestions, :third_party_engine, "")
-      timeout = Keyword.get(@suggestions, :timeout, 5000)
-      limit = Keyword.get(@suggestions, :limit, 23)
+    suggestions = Pleroma.Config.get(:suggestions)
 
-      host =
-        Application.get_env(:pleroma, Pleroma.Web.Endpoint)
-        |> Keyword.get(:url)
-        |> Keyword.get(:host)
+    if Keyword.get(suggestions, :enabled, false) do
+      api = Keyword.get(suggestions, :third_party_engine, "")
+      timeout = Keyword.get(suggestions, :timeout, 5000)
+      limit = Keyword.get(suggestions, :limit, 23)
+
+      host = Pleroma.Config.get([Pleroma.Web.Endpoint, :url, :host])
 
       user = user.nickname
       url = String.replace(api, "{{host}}", host) |> String.replace("{{user}}", user)
 
-      with {:ok, %{status_code: 200, body: body}} <-
-             @httpoison.get(url, [], timeout: timeout, recv_timeout: timeout),
+      with {:ok, %{status: 200, body: body}} <-
+             @httpoison.get(
+               url,
+               [],
+               adapter: [
+                 timeout: timeout,
+                 recv_timeout: timeout
+               ]
+             ),
            {:ok, data} <- Jason.decode(body) do
         data2 =
           Enum.slice(data, 0, limit)
@@ -1190,7 +1251,22 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
-  def filters(conn, _) do
-    json(conn, [])
+  def try_render(conn, renderer, target, params)
+      when is_binary(target) do
+    res = render(conn, renderer, target, params)
+
+    if res == nil do
+      conn
+      |> put_status(501)
+      |> json(%{error: "Can't display this activity"})
+    else
+      res
+    end
+  end
+
+  def try_render(conn, _, _, _) do
+    conn
+    |> put_status(501)
+    |> json(%{error: "Can't display this activity"})
   end
 end

lib/pleroma/web/mastodon_api/mastodon_socket.ex

@@ -5,15 +5,15 @@ defmodule Pleroma.Web.MastodonAPI.MastodonSocket do
   alias Pleroma.{User, Repo}
 
   transport(
-    :streaming,
+    :websocket,
     Phoenix.Transports.WebSocket.Raw,
     # We never receive data.
     timeout: :infinity
   )
 
-  def connect(params, socket) do
-    with token when not is_nil(token) <- params["access_token"],
-         %Token{user_id: user_id} <- Repo.get_by(Token, token: token),
+  @spec connect(params :: map(), Phoenix.Socket.t()) :: {:ok, Phoenix.Socket.t()} | :error
+  def connect(%{"access_token" => token} = params, socket) do
+    with %Token{user_id: user_id} <- Repo.get_by(Token, token: token),
          %User{} = user <- Repo.get(User, user_id),
          stream
          when stream in [
@@ -26,21 +26,43 @@ defmodule Pleroma.Web.MastodonAPI.MastodonSocket do
                 "list",
                 "hashtag"
               ] <- params["stream"] do
-      topic = if stream == "list", do: "list:#{params["list"]}", else: stream
-      socket_stream = if stream == "hashtag", do: "hashtag:#{params["tag"]}", else: stream
+      topic =
+        case stream do
+          "hashtag" -> "hashtag:#{params["tag"]}"
+          "list" -> "list:#{params["list"]}"
+          _ -> stream
+        end
 
       socket =
         socket
         |> assign(:topic, topic)
         |> assign(:user, user)
 
-      Pleroma.Web.Streamer.add_socket(socket_stream, socket)
+      Pleroma.Web.Streamer.add_socket(topic, socket)
       {:ok, socket}
     else
       _e -> :error
     end
   end
 
+  def connect(%{"stream" => stream} = params, socket)
+      when stream in ["public", "public:local", "hashtag"] do
+    topic =
+      case stream do
+        "hashtag" -> "hashtag:#{params["tag"]}"
+        _ -> stream
+      end
+
+    socket =
+      socket
+      |> assign(:topic, topic)
+
+    Pleroma.Web.Streamer.add_socket(topic, socket)
+    {:ok, socket}
+  end
+
+  def connect(_params, _socket), do: :error
+
   def id(_), do: nil
 
   def handle(:text, message, _state) do

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -10,14 +10,14 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
     render_many(users, AccountView, "account.json", opts)
   end
 
-  def render("account.json", %{user: user}) do
+  def render("account.json", %{user: user} = opts) do
     image = User.avatar_url(user) |> MediaProxy.url()
     header = User.banner_url(user) |> MediaProxy.url()
     user_info = User.user_info(user)
-    bot = (user.info["source_data"]["type"] || "Person") in ["Application", "Service"]
+    bot = (user.info.source_data["type"] || "Person") in ["Application", "Service"]
 
     emojis =
-      (user.info["source_data"]["tag"] || [])
+      (user.info.source_data["tag"] || [])
       |> Enum.filter(fn %{"type" => t} -> t == "Emoji" end)
       |> Enum.map(fn %{"icon" => %{"url" => url}, "name" => name} ->
         %{
@@ -29,10 +29,12 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       end)
 
     fields =
-      (user.info["source_data"]["attachment"] || [])
+      (user.info.source_data["attachment"] || [])
       |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
       |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
 
+    bio = HTML.filter_tags(user.bio, User.html_filter_policy(opts[:for]))
+
     %{
       id: to_string(user.id),
       username: username_from_nickname(user.nickname),
@@ -43,7 +45,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       followers_count: user_info.follower_count,
       following_count: user_info.following_count,
       statuses_count: user_info.note_count,
-      note: HTML.filter_tags(user.bio) || "",
+      note: bio || "",
       url: user.ap_id,
       avatar: image,
       avatar_static: image,
@@ -56,6 +58,11 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
         note: "",
         privacy: user_info.default_scope,
         sensitive: false
+      },
+
+      # Pleroma extension
+      pleroma: %{
+        tags: user.tags
       }
     }
   end
@@ -70,6 +77,15 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
   end
 
   def render("relationship.json", %{user: user, target: target}) do
+    follow_activity = Pleroma.Web.ActivityPub.Utils.fetch_latest_follow(user, target)
+
+    requested =
+      if follow_activity do
+        follow_activity.data["state"] == "pending"
+      else
+        false
+      end
+
     %{
       id: to_string(target.id),
       following: User.following?(user, target),
@@ -77,9 +93,10 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       blocking: User.blocks?(user, target),
       muting: false,
       muting_notifications: false,
-      requested: false,
+      requested: requested,
       domain_blocking: false,
-      showing_reblogs: false
+      showing_reblogs: false,
+      endorsed: false
     }
   end
 

lib/pleroma/web/mastodon_api/views/mastodon_view.ex

@@ -1,5 +1,4 @@
 defmodule Pleroma.Web.MastodonAPI.MastodonView do
   use Pleroma.Web, :view
   import Phoenix.HTML
-  import Phoenix.HTML.Form
 end

lib/pleroma/web/mastodon_api/views/push_subscription_view.ex

@@ -0,0 +1,11 @@
+defmodule Pleroma.Web.MastodonAPI.PushSubscriptionView do
+  use Pleroma.Web, :view
+
+  def render("push_subscription.json", %{subscription: subscription}) do
+    %{
+      id: to_string(subscription.id),
+      endpoint: subscription.endpoint,
+      alerts: Map.get(subscription.data, "alerts")
+    }
+  end
+end