I'm tring to write a basic listener for kernel.request event in Symfony 2. Service definition is pretty simple and annotations come from JMSDiExtraBundle.
The problems is that $context->getToken() is always null even the user is fully authenticated:
/**
* #Service("request.set_messages_count_listener")
*
*/
class RequestListener
{
/**
* #var \Symfony\Component\DependencyInjection\ContainerInterface
*/
private $container;
/**
* #InjectParams({"container" = #Inject("service_container")})
*
*/
public function __construct(ContainerInterface $container)
{
$this->container = $container;
}
/**
* #Observe("kernel.request", priority = 255)
*/
public function onKernelRequest(GetResponseEvent $event)
{
$context = $this->container->get('security.context');
var_dump($context->getToken()); die();
}
}
I think my security setup is working fine. What could be the problem then?
secured_area:
pattern: ^/app/
switch_user: true
form_login:
check_path: /app/login_check
login_path: /app/login
default_target_path: /app/dashboard
always_use_default_target_path: true
logout:
path: /demo/secured/logout # TODO
target: /demo/ # TODO
access_control:
- { path: ^/app/login, roles: IS_AUTHENTICATED_ANONYMOUSLY }
- { path: ^/app/users, roles: ROLE_MNG_USERS }
- { path: ^/app/messages, roles: ROLE_MNG_USERS }
- { path: ^/app/roles, roles: ROLE_MNG_PACKAGES_FEATURES }
- { path: ^/app/packages, roles: ROLE_MNG_PACKAGES_FEATURES }
- { path: ^/app/, roles: ROLE_USER }
With priority = 255, your listener is called BEFORE the security firewall (priority = 8, look here).
Try to change your priority.
Related
When using Spring Cloud Gateway (v3.1.3), how would one go about defining a default filter to perform retries for all routes, but then disable it for individual routes? I would like something as intuitive as this:
spring:
cloud:
gateway:
default-filters:
- name: Retry
args:
retries: 3
statuses: BAD_GATEWAY
methods: GET,POST,PUT,DELETE
backoff:
firstBackoff: 10ms
maxBackoff: 50ms
factor: 2
basedOnPreviousValue: false
routes:
- id: retry_disabled
uri: http://localhost:8080/retry_disabled
filters:
- name: Retry
args:
retries: 0
- id: retry_enabled
uri: http://localhost:8080/retry_enabled
I see in the RetryGatewayFilterFactory class that the RetryConfig.validate() method will fail when the number of retries is less than 1 or the other config options are not defined properly:
public void validate() {
Assert.isTrue(this.retries > 0, "retries must be greater than 0");
Assert.isTrue(!this.series.isEmpty() || !this.statuses.isEmpty() || !this.exceptions.isEmpty(),
"series, status and exceptions may not all be empty");
Assert.notEmpty(this.methods, "methods may not be empty");
if (this.backoff != null) {
this.backoff.validate();
}
}
Edit: I'm considering to implement it like this in code:
#Bean
public Function<GatewayFilterSpec, UriSpec> defaultRetryGatewayFilter() {
return gatewayFilterSpec -> gatewayFilterSpec
.retry(retryConfig -> {
RetryGatewayFilterFactory.BackoffConfig backoffConfig = new RetryGatewayFilterFactory.BackoffConfig();
backoffConfig.setFirstBackoff(Duration.ofMillis(10));
backoffConfig.setMaxBackoff(Duration.ofMillis(50));
backoffConfig.setFactor(2);
backoffConfig.setBasedOnPreviousValue(false);
retryConfig
.setRetries(3)
.allMethods()
.setSeries(HttpStatus.Series.SERVER_ERROR)
.setStatuses(HttpStatus.BAD_GATEWAY)
.setBackoff(backoffConfig);
});
}
#Bean
public RouteLocator routes(RouteLocatorBuilder builder, Function<GatewayFilterSpec, UriSpec> defaultRetryGatewayFilter) {
return builder.routes()
.route("retry_enabled", r -> r
.path("/retry_enabled")
.filters(defaultRetryGatewayFilter)
.uri("lb://foo"))
.route("retry_disabled", r -> r
.path("/retry_disabled")
// not retryable
.uri("lb://foo"))
.build();
}
Will the singleton defaultRetryGatewayFilter be thread safe?
There is A LOT of similar quesions, some of them have validated answers, but here I am and none of them worked.
My use case is pretty simple:
My users App\Client\common\Entities\User belong to a customer App\Client\common\Entities\Customer.
App\Client\common\Entities\User also inherits FOS\UserBundle\Model\User which contains the holy property "email"
I want to serialize all my customers AND their users (including their mail). Jms works pretty well except i can not access properties from the FOS\UserBundle\Model\User class.
following this answer here is what I have now.
jms_serializer.yml
jms_serializer:
#blablablaa....
metadata:
auto_detection: true
directories:
App:
namespace_prefix: 'App\Client'
path: '%kernel.project_dir%/serializer'
FOSUB:
namespace_prefix: 'FOS\UserBundle'
path: '%kernel.project_dir%/serializer'
serializer/App.Client.common.Entities.User.yml :
App\Client\common\Entities\User:
exclusion_policy: ALL
properties:
surname:
expose: true
exclude: false
groups: [export]
serializer/Model.User.yml:
FOS\UserBundle\Model\User:
exclusion_policy: ALL
properties:
email:
expose: true
exclude: false
groups: [export]
src/Command/DeploySyncUsersCommand.php:
protected function execute(InputInterface $input, OutputInterface $output)
{
$users = $this->customerRepository->findAll(); //this is an array of Customer
$context = new SerializationContext();
$context->setGroups(['export']);
$serializer = SerializerBuilder::create()->build();
$json = $serializer->serialize($users, 'json', $context);
// do something with json
}
Everything works fine except the json does NOT contain email or any FOSUser\User data.
Also something interesting is that I can write anything (even invalid yml) in the App.Client.common.Entities.User.yml and Model.User.yml files, I'm able to clear the cache with no errors. I have errors when I write invalid yml in jms_serializer.yml
Ok I've been able to solve this using dependency injection instead of building the serializer
/**
* #var SerializerInterface
*/
private SerializerInterface $serializer;
/**
* #var string
*/
private string $reportAnalysisUrl;
public function __construct(SerializerInterface $serialzer, $reportAnalysis)
{
$this->serializer = $serialzer;
parent::__construct();
}
protected function execute(InputInterface $input, OutputInterface $output)
{
$users = $this->customerRepository->findAll();
$context = new SerializationContext();
$context->setGroups(['export']);
$json = $this->serializer->serialize($users, 'json', $context);
//do something with json
}
so now my yml files aren't ignored anymore
I am using the Stormcrawler with the Elasticsearch example and no pages are shown with the FETCHED status in Kibana while crawling the webpage http://books.toscrape.com/
Still on the console the webpages appear to be fetched and parsed
48239 [Thread-26-fetcher-executor[3 3]] INFO c.d.s.b.FetcherBolt - [Fetcher #3] Threads : 0 queues : 1 in_queues : 1
48341 [FetcherThread #7] INFO c.d.s.b.FetcherBolt - [Fetcher #3] Fetched http://books.toscrape.com/catalogue/category/books_1/index.html with status 200 in msec 86
48346 [Thread-46-parse-executor[5 5]] INFO c.d.s.b.JSoupParserBolt - Parsing : starting http://books.toscrape.com/catalogue/category/books_1/index.html
48362 [Thread-46-parse-executor[5 5]] INFO c.d.s.b.JSoupParserBolt - Parsed http://books.toscrape.com/catalogue/category/books_1/index.html in 13 msec
Also it seems that the index of Elasticsearch gets some items even though these have no title
Screenshot of Kibana
I expanded the com.digitalpebble.stormcrawler.elasticsearch.bolt.IndexerBolt to also store the metadata of a web page in a local file and it seems like it does not get any tuples at all. Since the IndexerBolt also marks the status of a url as FETCHED that would explain the mentioned observation in Kibana.
Is there any explanation for this behaviour? I already reverted the crawler configuration to the standard, except the index bolt in crawler.flux to run my class.
The Topology Configuration:
name: "crawler"
includes:
- resource: true
file: "/crawler-default.yaml"
override: false
- resource: false
file: "es-conf.yaml"
override: true
spouts:
- id: "spout"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout"
parallelism: 10
bolts:
- id: "partitioner"
className: "com.digitalpebble.stormcrawler.bolt.URLPartitionerBolt"
parallelism: 1
- id: "fetcher"
className: "com.digitalpebble.stormcrawler.bolt.FetcherBolt"
parallelism: 1
- id: "sitemap"
className: "com.digitalpebble.stormcrawler.bolt.SiteMapParserBolt"
parallelism: 1
- id: "parse"
className: "com.digitalpebble.stormcrawler.bolt.JSoupParserBolt"
parallelism: 1
- id: "index"
className: "de.hpi.bpStormcrawler.IndexerBolt"
parallelism: 1
- id: "status"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt"
parallelism: 1
- id: "status_metrics"
className: "com.digitalpebble.stormcrawler.elasticsearch.metrics.StatusMetricsBolt"
parallelism: 1
streams:
- from: "spout"
to: "partitioner"
grouping:
type: SHUFFLE
- from: "spout"
to: "status_metrics"
grouping:
type: SHUFFLE
- from: "partitioner"
to: "fetcher"
grouping:
type: FIELDS
args: ["url"]
- from: "fetcher"
to: "sitemap"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "sitemap"
to: "parse"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "parse"
to: "index"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "fetcher"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "sitemap"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "parse"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "index"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
The reconfigured IndexerBolt
package de.hpi.bpStormcrawler;
/**
* Licensed to DigitalPebble Ltd under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* DigitalPebble licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import static com.digitalpebble.stormcrawler.Constants.StatusStreamName;
import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
import java.io.*;
import java.util.Iterator;
import java.util.Map;
import org.apache.storm.metric.api.MultiCountMetric;
import org.apache.storm.task.OutputCollector;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
import org.elasticsearch.action.DocWriteRequest;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.digitalpebble.stormcrawler.Metadata;
import com.digitalpebble.stormcrawler.elasticsearch.ElasticSearchConnection;
import com.digitalpebble.stormcrawler.indexing.AbstractIndexerBolt;
import com.digitalpebble.stormcrawler.persistence.Status;
import com.digitalpebble.stormcrawler.util.ConfUtils;
/**
* Sends documents to ElasticSearch. Indexes all the fields from the tuples or a
* Map <String,Object> from a named field.
*/
#SuppressWarnings("serial")
public class IndexerBolt extends AbstractIndexerBolt {
private static final Logger LOG = LoggerFactory
.getLogger(IndexerBolt.class);
private static final String ESBoltType = "indexer";
static final String ESIndexNameParamName = "es.indexer.index.name";
static final String ESDocTypeParamName = "es.indexer.doc.type";
private static final String ESCreateParamName = "es.indexer.create";
private OutputCollector _collector;
private String indexName;
private String docType;
// whether the document will be created only if it does not exist or
// overwritten
private boolean create = false;
File indexFile;
private MultiCountMetric eventCounter;
private ElasticSearchConnection connection;
#SuppressWarnings({ "unchecked", "rawtypes" })
#Override
public void prepare(Map conf, TopologyContext context,
OutputCollector collector) {
super.prepare(conf, context, collector);
_collector = collector;
indexName = ConfUtils.getString(conf, IndexerBolt.ESIndexNameParamName,
"fetcher");
docType = ConfUtils.getString(conf, IndexerBolt.ESDocTypeParamName,
"doc");
create = ConfUtils.getBoolean(conf, IndexerBolt.ESCreateParamName,
false);
try {
connection = ElasticSearchConnection
.getConnection(conf, ESBoltType);
} catch (Exception e1) {
LOG.error("Can't connect to ElasticSearch", e1);
throw new RuntimeException(e1);
}
this.eventCounter = context.registerMetric("ElasticSearchIndexer",
new MultiCountMetric(), 10);
indexFile = new File("/Users/jonaspohlmann/code/HPI/BP/stormCrawlerSpike/spikeStormCrawler2/index.log");
}
#Override
public void cleanup() {
if (connection != null)
connection.close();
}
#Override
public void execute(Tuple tuple) {
String url = tuple.getStringByField("url");
// Distinguish the value used for indexing
// from the one used for the status
String normalisedurl = valueForURL(tuple);
Metadata metadata = (Metadata) tuple.getValueByField("metadata");
String text = tuple.getStringByField("text");
//BP: added Content Field
String content = new String(tuple.getBinaryByField("content"));
boolean keep = filterDocument(metadata);
if (!keep) {
eventCounter.scope("Filtered").incrBy(1);
// treat it as successfully processed even if
// we do not index it
_collector.emit(StatusStreamName, tuple, new Values(url, metadata,
Status.FETCHED));
_collector.ack(tuple);
return;
}
try {
XContentBuilder builder = jsonBuilder().startObject();
// display text of the document?
if (fieldNameForText() != null) {
builder.field(fieldNameForText(), trimText(text));
}
// send URL as field?
if (fieldNameForURL() != null) {
builder.field(fieldNameForURL(), normalisedurl);
}
// which metadata to display?
Map<String, String[]> keyVals = filterMetadata(metadata);
Iterator<String> iterator = keyVals.keySet().iterator();
while (iterator.hasNext()) {
String fieldName = iterator.next();
String[] values = keyVals.get(fieldName);
if (values.length == 1) {
builder.field(fieldName, values[0]);
try {
saveStringToFile(indexFile, fieldName + "\t" + values[0]);
} catch (IOException e) {
e.printStackTrace();
}
} else if (values.length > 1) {
builder.array(fieldName, values);
}
}
builder.endObject();
String sha256hex = org.apache.commons.codec.digest.DigestUtils
.sha256Hex(normalisedurl);
IndexRequest indexRequest = new IndexRequest(indexName, docType,
sha256hex).source(builder);
DocWriteRequest.OpType optype = DocWriteRequest.OpType.INDEX;
if (create) {
optype = DocWriteRequest.OpType.CREATE;
}
indexRequest.opType(optype);
connection.getProcessor().add(indexRequest);
eventCounter.scope("Indexed").incrBy(1);
_collector.emit(StatusStreamName, tuple, new Values(url, metadata,
Status.FETCHED));
_collector.ack(tuple);
} catch (IOException e) {
LOG.error("Error sending log tuple to ES", e);
// do not send to status stream so that it gets replayed
_collector.fail(tuple);
}
}
private void saveStringToFile(File file, String stringToWrite) throws IOException {
String pathName = file.getPath();
File folder = file.getParentFile();
if (!folder.exists() && !folder.mkdirs()) {
throw new IOException("Couldn't create the storage folder: " + folder.getAbsolutePath() + " does it already exist ?");
}
try (PrintWriter out = new PrintWriter(new FileOutputStream(file, true))) {
out.append(stringToWrite + '\n');
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
}
Have you merged all your configs i.e generic SC + specific ES one into a single es-conf.yaml? If not then your Flux file is probably missing
- resource: false
file: "crawler-conf.yaml"
override: true
where the indexer config typically looks like:
indexer.url.fieldname: "url"
indexer.text.fieldname: "content"
indexer.canonical.name: "canonical"
indexer.md.mapping:
- parse.title=title
- parse.keywords=keywords
- parse.description=description
- domain=domain
Not having any md mappings defined would explain why your modified indexer does not write to the files and why the index contains urls but no additional fields.
Please note that the 'index' index (excuse the terminology) does not contain the status of the URL. See https://stackoverflow.com/a/49316316/432844 for an explanation of status vs index.
I'am trying to using the FOSUserBundle with Symfony 3.0.9.
After doing the tutorial here I was able to register a user and login with it, yesterday. The problem was that I couldn't login this morning with the same credentials. After a some tests I noticed that it's not possible to login with the user, when I closed the window.
So i have to delete the user from database and create a new one.
Can anybody help me out to fix this problem?
Surely a user should be able to login, logout and close the brwoser without a logout.
My SecurityController.php:
class SecurityController extends Controller
{
/**
* #Route("/login", name="login")
*/
public function loginAction(Request $request)
{
$authenticationUtils = $this->get('security.authentication_utils');
// get the login error if there is one
$error = $authenticationUtils->getLastAuthenticationError();
// last username entered by the user
$lastUsername = $authenticationUtils->getLastUsername();
return $this->render(
'security/login.html.twig',
array(
// last username entered by the user
'last_username' => $lastUsername,
'error' => $error,
)
);
}
}
security.yml:
# To get started with security, check out the documentation:
# http://symfony.com/doc/current/book/security.html
security:
encoders:
FOS\UserBundle\Model\UserInterface: bcrypt
role_hierarchy:
ROLE_ADMIN: ROLE_USER
ROLE_SUPER_ADMIN: ROLE_ADMIN
providers:
fos_userbundle:
id: fos_user.user_provider.username
firewalls:
# disables authentication for assets and the profiler, adapt it according to your needs
dev:
pattern: ^/(_(profiler|wdt)|css|images|js)/
security: false
main:
pattern: ^/
form_login:
provider: fos_userbundle
csrf_token_generator: security.csrf.token_manager
logout: true
anonymous: true
access_control:
- { path: ^/login$, role: IS_AUTHENTICATED_ANONYMOUSLY }
- { path: ^/register, role: IS_AUTHENTICATED_ANONYMOUSLY }
- { path: ^/resetting, role: IS_AUTHENTICATED_ANONYMOUSLY }
- { path: ^/admin/, role: ROLE_ADMIN }
At least my user entity in database:
The problem was that the session was set by /register and my /login never worked.
With the standard FOSUser Login it worked without problems.
I have a problem with swift mailer and monolog on Symfony 3.0.2:
FatalThrowableError in appDevDebugProjectContainer.php line 4963:
Type error: Argument 1 passed to SymfonyBundleMonologBundleSwiftMailerMessageFactory_0000000079e53f2b00000001716bb61a50d0bc982eb9e83148fbcc469ab36a58::__construct() must be an instance of Swift_Mailer, instance of Closure given, called in /Users/Romain/Sites/var/cache/dev/appDevDebugProjectContainer.php on line 4043
# Swiftmailer Configuration config.yml
swiftmailer:
transport: "%mailer_transport%"
host: "%mailer_host%"
username: "%mailer_user%"
password: "%mailer_password%"
spool: { type: memory }
#Monolog config_prod.yml
monolog:
handlers:
main:
type: fingers_crossed
action_level: critical
handler: grouped
grouped:
type: group
members: [streamed, buffered]
streamed:
type: stream
path: "%kernel.logs_dir%/%kernel.environment%.log"
level: debug
buffered:
type: buffer
handler: swift
swift:
type: swift_mailer
from_email: no-reply#email.com
to_email: email#email.com
subject: "Subject"
level: debug
An extract appDevDebugProjectContainer.php on line 4963
/**
* {#inheritDoc}
*/
public function __construct(\Swift_Mailer $mailer, $fromEmail, $toEmail, $subject, $contentType = null)
{
static $reflection;
if (! $this->valueHolder56d41e956b1f5441039037) {
$reflection = $reflection ?: new \ReflectionClass('Symfony\\Bundle\\MonologBundle\\SwiftMailer\\MessageFactory');
$this->valueHolder56d41e956b1f5441039037 = $reflection->newInstanceWithoutConstructor();
\Closure::bind(function (\Symfony\Bundle\MonologBundle\SwiftMailer\MessageFactory $this) {
unset($this->mailer, $this->fromEmail, $this->toEmail, $this->subject, $this->contentType);
}, $this, 'Symfony\\Bundle\\MonologBundle\\SwiftMailer\\MessageFactory')->__invoke($this);
}
$this->valueHolder56d41e956b1f5441039037->__construct($mailer, $fromEmail, $toEmail, $subject, $contentType);
}
An extract appDevDebugProjectContainer.php on line 4043
/**
* Gets the 'monolog.handler.swift.mail_message_factory' service.
*
* This service is shared.
* This method always returns the same instance of the service.
*
* This service is private.
* If you want to be able to request this service from the container directly,
* make it public, otherwise you might end up with broken code.
*
* #param bool $lazyLoad whether to try lazy-loading the service with a proxy
*
* #return \Symfony\Bundle\MonologBundle\SwiftMailer\MessageFactory A Symfony\Bundle\MonologBundle\SwiftMailer\MessageFactory instance.
*/
public function getMonolog_Handler_Swift_MailMessageFactoryService($lazyLoad = true)
{
if ($lazyLoad) {
return $this->services['monolog.handler.swift.mail_message_factory'] = new SymfonyBundleMonologBundleSwiftMailerMessageFactory_0000000057f95edf000000015dd8d44e50d0bc982eb9e83148fbcc469ab36a58(
function (&$wrappedInstance, \ProxyManager\Proxy\LazyLoadingInterface $proxy) {
$wrappedInstance = $this->getMonolog_Handler_Swift_MailMessageFactoryService(false);
$proxy->setProxyInitializer(null);
return true;
}
);
}
return new \Symfony\Bundle\MonologBundle\SwiftMailer\MessageFactory($this->get('swiftmailer.mailer.default'), 'contact#domaine.com', array(0 => 'error#domaine.com'), 'Une erreur critique est survenue', NULL);
}
Sends him of e-mail with swiftmailer only work.
I already have this configuration with the same environment but symfony 2.7 and that works.
And this configuration works on a wamp (php7) but not on my environement OSX and server Linux ...
Thank you for your help
fix with symfony 3.0.3 and monolog 1.18